Papers with machine learning
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop) (2025.naacl-srw)
Copied to clipboard
| Challenge: | Philip is a member of the Association for Computational Linguistics and is pursuing his PhD in computational linguistics. |
| Approach: | Philip is a member of the Association for Computational Linguistics and is pursuing a PhD in computational linguistics. |
| Outcome: | Philip is a member of the Association for Computational Linguistics and is pursuing two PhDs in computational linguistics and cognitive neuroscience. |
NLP+Vis: NLP Meets Visualization (2023.emnlp-tutorial)
Copied to clipboard
| Challenge: | This tutorial will introduce NLP+Vis with a focus on two main threads of work: NLP for Vis and Vis for NLP. |
| Approach: | tutorial will introduce NLP+Vis with a focus on two main threads of work . overview of research topics on combining NLP and Vis techniques will be covered . |
| Outcome: | The tutorial will introduce NLP+Vis with a focus on two main threads of work . it will provide an overview of research topics on combining NLP and Vis techniques . |
Chasing the Tail with Domain Generalization: A Case Study on Frequency-Enriched Datasets (2022.aacl-main)
Copied to clipboard
| Challenge: | In academic research, natural language understanding tasks are typically defined by creating annotated datasets in which each utterance is encountered once. |
| Approach: | They propose a method that explicitly uses utterance frequency in training data to learn models that are more robust to unknown distributions. |
| Outcome: | The proposed approach shows up to 7.02% relative improvement over baselines on the tail data. |
Multi-Dialectal Representation Learning of Sinitic Phonology (2023.acl-srw)
Copied to clipboard
| Challenge: | In Sinitic Historical Phonology, notable tasks that could benefit from machine learning include the comparison of dialects and reconstruction of proto-languages systems. |
| Approach: | They propose to use a knowledge graph to obtain multi-dialectal representations of Sinitic syllables by using unsupervised clustering techniques and the BoxE technique from knowledge base learning. |
| Outcome: | The proposed representations capture phonemic contrast from the input dialects and can be used to infer Middle Chinese labels. |
CodeAnno: Extending WebAnno with Hierarchical Document Level Annotation and Automation (2023.eacl-demo)
Copied to clipboard
Florian Schneider, Seid Muhie Yimam, Fynn Petersen-frey, Gerret Von Nordheim, Katharina Kleinen-von K”onigsl”ow, Chris Biemann
| Challenge: | WebAnno focuses on document-level annotation, which is complicated. |
| Approach: | They propose to create hierarchical codebooks that allow to move and sort categories in the hierarchy. |
| Outcome: | The proposed system is based on the existing WebAnno annotation tools and is compatible with existing spreadsheet applications. |
lingvis.io - A Linguistic Visual Analytics Framework (P19-3)
Copied to clipboard
Mennatallah El-Assady, Wolfgang Jentner, Fabian Sperrle, Rita Sevastjanova, Annette Hautli-Janisz, Miriam Butt, Daniel Keim
| Challenge: | Using a modular framework, linguistic visual analytics applications can be rapidly prototypized using a web-based framework. |
| Approach: | They propose a modular framework for rapid prototyping of linguistic, web-based, visual analytics applications. |
| Outcome: | The proposed framework supports rapid prototyping of linguistic, web-based, visual analytics applications. |
Meta Learning and Its Applications to Natural Language Processing (2021.acl-tutorials)
Copied to clipboard
| Challenge: | Meta-learning is a new technique that aims to learn better learning algorithms, including better parameter initialization, optimization strategy, network architecture, distance metrics, and beyond. |
| Approach: | This tutorial introduces Meta-learning approaches and the theory behind them, and then reviews the works of applying this technology to NLP problems. |
| Outcome: | This tutorial will introduce Meta-learning approaches and the theory behind them, and then review the works of applying this technology to NLP problems. |
Rumor Detection on Twitter Using Multiloss Hierarchical BiLSTM with an Attenuation Factor (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing models to classify rumors have low precision and are time consuming. |
| Approach: | They propose a multiloss hierarchical biLSTM model with an attenuation factor that can extract deep information from limited quantities of text. |
| Outcome: | The proposed model can extract deep information from limited quantities of text. |
Explanation in the Era of Large Language Models (2024.naacl-tutorials)
Copied to clipboard
| Challenge: | Explanation has long been a part of communication, where humans use language to elucidate each other and transmit information about mechanisms of events. |
| Approach: | They review the opportunities and challenges of explanations in the era of large language models and examine how they can be used to generate explanations. |
| Outcome: | The proposed methods are based on the models of large language models (LLMs) and their opaque nature. |
Bias and Fairness in Natural Language Processing (D19-2)
Copied to clipboard
| Challenge: | a tutorial will review the history of bias and fairness studies in machine learning and language processing . |
| Approach: | This tutorial reviews the history of bias and fairness studies in machine learning and language processing . it presents recent community effort to quantify and mitigat bias in natural language processing models . |
| Outcome: | This tutorial reviews the history of bias and fairness studies in machine learning and language processing . it aims to quantify and mitigate bias in natural language processing models for a wide spectrum of tasks . |
LLM-driven Instruction Following: Progresses and Concerns (2023.emnlp-tutorial)
Copied to clipboard
| Challenge: | a tutorial on task instruction is aimed at researchers and practitioners interested in NLP generalization . labeled examples are unlikely to be available in large numbers or do not exist . |
| Approach: | This tutorial will examine the progress of natural language processing (NLP) using labeled examples. authors propose that task instructions act as a novel resource for supervision. |
| Outcome: | This tutorial aims to answer questions about instruction-driven NLP . it focuses on the use of task instructions in a low-shot scenario . |
Deep Bayesian Learning and Understanding (C18-3)
Copied to clipboard
| Challenge: | COLING 2018 is a conference for researchers and practitioners working on machine learning and deep learning. |
| Approach: | a tutorial on machine learning and deep learning will be presented at COLING 2018 . the tutorial will focus on statistical models, deep neural networks, sequential learning and natural language understanding . |
| Outcome: | This tutorial will present the latest advances in deep Bayesian and sequential learning at COLING 2018 . |
Representations of Meaning in Neural Networks for NLP: a Thesis Proposal (2021.naacl-srw)
Copied to clipboard
| Challenge: | Neural networks are the state-of-the-art method of machine learning for many problems in NLP. |
| Approach: | They propose to examine the distribution of meaning in the vector space representation of words in neural networks trained for NLP tasks. |
| Outcome: | The proposed method would be compatible with distributional hypothesis, structuralism, and semantic holism. |
LUX (Linguistic aspects Under eXamination): Discourse Analysis for Automatic Fake News Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Automated fact-checking is time-consuming and cannot scale due to a lack of suitable training data. |
| Approach: | They propose to use a dataset to automatically check facts and a text classifier to infer the likelihood of the input being a piece of fake-news. |
| Outcome: | The proposed dataset VERITAS and LUX use linguistic analysis to infer the likelihood of the input being a piece of fake-news. |
Knowledge Distillation for Language Models (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a teacher to a student . this tutorial will cover topics ranging from LLM sequence compression to LLM self-distillation . |
| Approach: | They propose to introduce intermediate-layer matching and prediction matching . they will then present advanced techniques such as reinforcement learning-based KD and multi-teacher distillation . |
| Outcome: | This tutorial aims to provide participants with a comprehensive understanding of the techniques and applications of knowledge distillation for language models. |
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System? (2020.tacl-1)
Copied to clipboard
| Challenge: | Data privacy is an important issue for “machine learning as a service” providers. |
| Approach: | They propose an attack on membership inference attacks using a sequence-to-sequence model and a machine translation dataset to investigate the feasibility of a privacy attack. |
| Outcome: | The proposed model can infer sentence-level membership from the output of the model, but it is difficult to infer it. |
Beyond Multiword Expressions: Processing Idioms and Metaphors (P18-5)
Copied to clipboard
| Challenge: | idioms and metaphors processing is a rapidly growing area in NLP, says dr. s. robertson . idiomatic idiomas are characteristic to all areas of human activity and to all types of discourse. |
| Approach: | This tutorial will provide attendees with a clear notion of idioms and metaphors . it will provide them with computational models of linguistic characteristics and methods . |
| Outcome: | This tutorial aims to provide attendees with a clear notion of the linguistic characteristics of idioms and metaphors . it outlines how to model idiomatic idiomes and their processing and what resources are available to support their use . |
Automatic Classification of Students on Twitter Using Simple Profile Information (2020.aacl-srw)
Copied to clipboard
| Challenge: | Existing models for age classification of students and non-students are restrictive and require access to many tweets. |
| Approach: | They propose a model which uses 3 tweet-content features to classify users as students or non-students. |
| Outcome: | The proposed model achieves an accuracy of 88.1% and an F1 score of .704 compared to previous models, which require access to many user tweets. |
Indirectly Supervised Natural Language Processing (2023.acl-tutorials)
Copied to clipboard
| Challenge: | a tutorial on indirect supervision addresses challenges in ML for NLP . conventional approaches to NLP use taskspecific labeled examples of a large volume . indirect supervision is useful for a wide range of NLP tasks, but it is not enough for decoders . |
| Approach: | This tutorial aims to address questions about indirect supervision in machine learning . authors discuss indirect supervision from T′ that handles T with outputs spanning from a moderate size to an open space . |
| Outcome: | This tutorial aims to answer questions about how to provide supervision for ML tasks . it will discuss indirect supervision from T′ that handles T with outputs spanning from a moderate size to an open space . |
Research on Task Discovery for Transfer Learning in Deep Neural Networks (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing deep neural network based machine learning models suffer from overfitting and are sensitive to noise and examples that are not available in training data. |
| Approach: | They propose to use a novel multi-task learner to implement deep neural network based transfer learning models that can be used to improve generalization. |
| Outcome: | The proposed model performs better on two NLP tasks and is more efficient on other areas of machine learning, including Bioinformatics and Computer Vision. |
Recognizing Multimodal Entailment (2021.acl-tutorials)
Copied to clipboard
Cesar Ilharco, Afsaneh Shirazi, Arjun Gopalan, Arsha Nagrani, Blaz Bratanic, Chris Bregler, Christina Funk, Felipe Ferreira, Gabriel Barcik, Gabriel Ilharco, Georg Osang, Jannis Bulian, Jared Frank, Lucas Smaira, Qin Cao, Ricardo Marino, Roma Patel, Thomas Leung, Vaiva Imbrasaite
| Challenge: | This tutorial introduces the multimodal entailment task for detecting semantic alignments . the task requires fine-grained understanding of visual and linguistic semantics questions . |
| Approach: | This tutorial introduces the multimodal entailment task to machine learning . it introduces a dataset for recognizing multimodal alignments . |
| Outcome: | This tutorial introduces the multimodal entailment task . it can be useful for detecting semantic alignments when a single modality alone is not enough . |
Low-resource Cross-lingual Event Type Detection via Distant Supervision with Minimal Effort (C18-1)
Copied to clipboard
| Challenge: | Currently, few or no language processing tools or resources exist for most languages . a problem is that there is not enough available training data even in resource-rich languages if the task is complex. |
| Approach: | They propose to use a bilingual dictionary to train machine learning in a resource-poor language . they also explore adversarial training of bilingual word representations . |
| Outcome: | The proposed approach gives similar performance in event-type detection tasks. |
DERE: A Task and Domain-Independent Slot Filling Framework for Declarative Relation Extraction (D18-2)
Copied to clipboard
| Challenge: | Comparability of models across tasks is lacking in most machine learning systems for natural language processing. |
| Approach: | They propose a framework for declarative specification and compilation of template-based information extraction that uses a generic specification language for the task and for data annotations in terms of spans and frames. |
| Outcome: | The proposed framework enables representation of a large variety of natural language processing tasks. |
Forum 4.0: An Open-Source User Comment Analysis Framework (2021.eacl-demos)
Copied to clipboard
Marlo Haering, Jakob Smedegaard Andersen, Chris Biemann, Wiebke Loosen, Benjamin Milde, Tim Pietz, Christian Stöcker, Gregor Wiedemann, Olaf Zukunft, Walid Maalej
| Challenge: | Using Forum 4.0, we analyze, aggregate, and visualize user comments based on labels defined by domain experts. |
| Approach: | They introduce an open-source framework to semi-automatically analyze, aggregate, and visualize user comments based on labels defined by domain experts. |
| Outcome: | The proposed framework can analyze, aggregate, and visualize user comments based on labels defined by domain experts. |
Attention Flows are Shapley Value Explanations (2021.acl-short)
Copied to clipboard
| Challenge: | Shapley Values are a popular type of explanation in machine learning, but leave-one-out and attention-based explanations still predominate in NLP. |
| Approach: | They propose to use attention flow to explain the importance of features, embeddings, and even neurons to explain credit assignment problems in cooperative game theory. |
| Outcome: | The proposed explanations can explain the importance of features, embeddings, and even neurons, but in NLP, leave-one-out and attention-based explanations still predominate. |
Automated Cross-language Intelligibility Analysis of Parkinson’s Disease Patients Using Speech Recognition Technologies (P19-2)
Copied to clipboard
| Challenge: | PD is the second most common neurodegenerative disorder after Alzheimers disease . speech impairments are one of the earliest manifestations in PD patients . |
| Approach: | They propose to analyze the speech signals of PD patients and healthy control subjects in three different languages: German, Spanish, and Czech. |
| Outcome: | The proposed model can discriminate between PD patients and HC subjects even when the language used for train and test is different. |
Supporting Complaints Investigation for Nursing and Midwifery Regulatory Agencies (2021.acl-demo)
Copied to clipboard
Piyawat Lertvittayakumjorn, Ivan Petej, Yang Gao, Yamuna Krishnamurthy, Anna Van Der Gaag, Robert Jago, Kostas Stathis
| Challenge: | Fig. 1 illustrates the major components and workflow of our proposed system to improve the efficiency of complaints investigation for nursing and midwifery regulators. |
| Approach: | They propose a decision support system that uses machine learning and natural language processing techniques to process complaints and predict their risk level. |
| Outcome: | The proposed system uses state-of-the-art machine learning and natural language processing techniques to process complaints and predict risk levels. |
Introducing the CURLICAT Corpora: Seven-language Domain Specific Annotated Corpora from Curated Sources (2022.lrec-1)
Copied to clipboard
Tamás Váradi, Bence Nyéki, Svetla Koeva, Marko Tadić, Vanja Štefanec, Maciej Ogrodniczuk, Bartłomiej Nitoń, Piotr Pęzik, Verginica Barbu Mititelu, Elena Irimia, Maria Mitrofan, Dan Tufiș, Radovan Garabík, Simon Krek, Andraž Repar
| Challenge: | The CURLICAT CEF Telecom project aims to collect and deeply annotate a set of large corpora from selected domains. |
| Approach: | They present the results of the CURLICAT CEF Telecom project . they propose to collect and deeply annotate a set of large corpora from selected domains . |
| Outcome: | The CURLICAT CEF Telecom project provides a set of large corpora from selected domains . the corporatized corporates are tokenized, lemmatized and morphologically analysed . |
A Survey on Automated Fact-Checking (2022.tacl-1)
Copied to clipboard
| Challenge: | Fact-checking is an essential task in journalism due to the speed with which information and misinformation can spread in the media ecosystem. |
| Approach: | They propose to use natural language processing to automate fact-checking by identifying common concepts and defining definitions. |
| Outcome: | The proposed method can predict the veracity of claims using natural language processing, machine learning, and databases. |
Automatic identification of writers’ intentions: Comparing different methods for predicting relationship goals in online dating profile texts (D19-55)
Copied to clipboard
| Challenge: | lexicon-based text analysis methods such as LIWC have been criticized by computational linguists for their lack of adaptability, but they have not been systematically compared with either human evaluations or machine learning approaches. |
| Approach: | They used a corpus of online dating profile texts to compare LIWC, machine learning, and a human baseline to assess their effectiveness on a relationship goal classification task. |
| Outcome: | The proposed methods were compared with a corpus of online dating profile texts and a human baseline. |
Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements (2022.emnlp-demos)
Copied to clipboard
Leandro Von Werra, Lewis Tunstall, Abhishek Thakur, Sasha Luccioni, Tristan Thrush, Aleksandra Piktus, Felix Marty, Nazneen Rajani, Victor Mustar, Helen Ngo
| Challenge: | Evaluation is a key part of machine learning, yet there is neo-tooling to support it . auxiliary techniques such as testing for significance, measuring statistical power, and auxiliary methods are not available in ML. |
| Approach: | They propose a set of tools to facilitate the evaluation of models and datasets in machine learning . they propose 'evaluation on the Hub' platform that enables large-scale evaluation of over 75,000 models . |
| Outcome: | The proposed tools can be used to evaluate models and datasets on the Hugging Face Hub. |
Veritas Annotator: Discovering the Origin of a Rumour (D19-66)
Copied to clipboard
| Challenge: | a growing number of fake news sites are used for spreading fake news . a lack of a reliable data set is limiting the use of machine learning in fact-checking . |
| Approach: | They propose a web application that can detect the origin of a rumour by identifying its source . |
| Outcome: | The proposed application can detect fake news claims with better accuracy than humans . |
RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs (2024.naacl-demo)
Copied to clipboard
| Challenge: | Recent advances in machine learning (ML) are attributed to large language models (LLMs), but their escalating memory requirements require developers to partition a large model to distribute it across multiple GPUs or TPUs. |
| Approach: | They propose a lightweight and user-friendly tool to automate distributed training and inference for LLMs and to simplify ML pipeline development. |
| Outcome: | The proposed tool automates distributed training and inference for LLMs, and simplifies ML pipeline development. |
Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours (2022.emnlp-demos)
Copied to clipboard
Eyal Shnarch, Alon Halfon, Ariel Gera, Marina Danilevsky, Yannis Katsis, Leshem Choshen, Martin Santillan Cooper, Dina Epelboim, Zheng Zhang, Dakuo Wang
| Challenge: | Label Sleuth is an open source system for labeling and creating text classifiers which does not require coding skills nor machine learning knowledge. |
| Approach: | *Label Sleuth* is an open source system for labeling and creating text classifiers which does not require coding skills nor machine learning knowledge. |
| Outcome: | *Label Sleuth* is an open source system for labeling and creating text classifiers. |
Machine Learning–Driven Language Assessment (2020.tacl-1)
Copied to clipboard
| Challenge: | Language proficiency tests are cumbersome to create and maintain, and items may be copied and leaked or simply used too often. |
| Approach: | They propose a method that uses machine learning and natural language processing to induce proficiency scales and linguistic models to estimate item difficulty directly for computer-adaptive testing. |
| Outcome: | The proposed method produces scores that are reliable and reliable while generating item banks large enough to satisfy security requirements. |
DataLab: A Platform for Data Analysis and Intervention (2022.acl-demo)
Copied to clipboard
Yang Xiao, Jinlan Fu, Weizhe Yuan, Vijay Viswanathan, Zhoumianze Liu, Yixin Liu, Graham Neubig, Pengfei Liu
| Challenge: | Existing tools and research focus on how to interpret and manipulate data, despite its crucial role in machine learning, . existing tools and researchers focus on systems on top of existing data, rather than how to use it. |
| Approach: | They propose a unified data-oriented platform that allows users to interactively analyze the characteristics of data and provides a standard interface for many data processing operations. |
| Outcome: | The proposed platform allows users to analyze the characteristics of data and provides a standardized interface so that many data processing operations can be provided within a single interface. |
Statistical Deficiency for Task Inclusion Estimation (2025.acl-long)
Copied to clipboard
Loïc Fosse, Frederic Bechet, Benoit Favre, Géraldine Damnati, Gwénolé Lecorvé, Maxime Darrin, Philippe Formont, Pablo Piantanida
| Challenge: | Tasks are central in machine learning, as they are the most natural objects to assess the capabilities of current models. |
| Approach: | They propose a theoretically grounded setup to define the notion of task and compute the inclusion between two tasks from a statistical deficiency point of view. |
| Outcome: | The proposed model estimates the degree of inclusion between tasks on synthetic data and reconstructs the classic NLP pipeline. |
Using Snomed to recognize and index chemical and drug mentions. (D19-57)
Copied to clipboard
| Challenge: | a new named entity extraction system is proposed for biological texts . the system is based on machine learning and deep learning . |
| Approach: | They propose a named entity extraction system based on machine learning and deep learning . they propose to map drug names in Spanish biomedical texts using Snomed . |
| Outcome: | The proposed system achieves 78% in the first sub-track and 72% in the second task. |
Representation and Generation of Machine Learning Test Functions (2024.eacl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been adopted for ML code generation but their implications are relatively unexplored. |
| Approach: | They examine the use of Large Language Models to extract representations of ML source code and tests to understand the semantic relationships between human-written tests and LLM-generated ones. |
| Outcome: | The proposed models can be used to extract representations of ML source code and tests and annotate them for usefulness, documentation, and correctness. |
Model-agnostic Methods for Text Classification with Inherent Noise (2020.coling-industry)
Copied to clipboard
| Challenge: | Text classification is a fundamental problem in natural language processing, but its performance relies on high-quality annotations. |
| Approach: | They propose to use model-agnostic methods to handle inherent noise in large scale text classification that can be easily incorporated into existing machine learning workflows with minimal interruption. |
| Outcome: | The proposed method outperforms baselines by up to 10% in classification accuracy while requiring no network modifications. |
Detecting Primary Progressive Aphasia (PPA) from Text: A Benchmarking Study (2026.findings-eacl)
Copied to clipboard
Ghofrane Merhbene, Fabian Lecron, Philippe Fortemps, Bradford C. Dickerson, Mascha Kurpicz-Briki, Neguine Rezaii
| Challenge: | Primary progressive aphasia (PPA) is a neurodegenerative disorder characterized by progressive language deficits as the primary symptom. |
| Approach: | They benchmarked the performance of traditional machine learning models with various feature extraction techniques, transformer-based models, and large language models (LLMs) they found that transformer-Based models exceeded chance-level performance in terms of balanced accuracy, while MLP using MentalBert’s embeddings achieved the highest accuracy. |
| Outcome: | The proposed models outperform chance-level models in terms of balanced accuracy while using MentalBert’s embeddings achieve the highest accuracy. |
Emotion Classification in a Resource Constrained Language Using Transformer-based Approach (2021.naacl-srw)
Copied to clipboard
| Challenge: | Existing methods to classify Bengali text into six basic emotions are infancy for resource-constrained languages like English, Arabic, Chinese and French. |
| Approach: | They propose a transformer-based technique to classify Bengali text into one of the six basic emotions: anger, fear, disgust, sadness, joy, and surprise. |
| Outcome: | The proposed technique outperforms all other techniques by achieving highest weighted f_1-score on the test data. |
Automated Screening of Antibacterial Nanoparticle Literature: Dataset Curation and Model Evaluation (2026.eacl-long)
Copied to clipboard
Alperen Ozturk, Şaziye Betül Özateş, Sophia Bahar Root, Angela Violi, Nicholas Kotov, J. Scott VanEpps, Emine Sumeyra Turali Emre
| Challenge: | Antimicrobial resistance is a growing global health threat, driving interest in nanoparticle-based alternatives to conventional antibiotics. |
| Approach: | They propose to use machine learning to classify scientific abstracts using inorganic nanoparticles with intrinsic antibacterial properties. |
| Outcome: | The proposed method distinguishes intrinsic antibacterial NPs from studies focusing on drug carriers or surface-bound applications. |
Towards Unified Representations of Knowledge Graph and Expert Rules for Machine Learning and Reasoning (2022.aacl-main)
Copied to clipboard
| Challenge: | Empirical study shows superiority of proposed method over time-tested knowledge-driven and data-driven methods. |
| Approach: | They propose a cognitive knowledge graph that unifies expert rules and relational facts as the substrate of machine learning and reasoning models. |
| Outcome: | Empirical results show the proposed method superior to time-tested methods . the proposed model can perform both learning and reasoning with labeled data . |
Marrying LLMs with Dynamic Forecasting: A Graph Mixture-of-expert Perspective (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent data-driven approaches often use graph neural networks (GNNs) to learn relationships in dynamical systems. |
| Approach: | They propose a framework which leverages large language models to enhance generalization capabilities of dynamical system modeling. |
| Outcome: | The proposed framework improves on existing methods and compares to baselines. |
Native Language Prediction from Gaze: a Reproducibility Study (2023.acl-srw)
Copied to clipboard
| Challenge: | Existing studies have shown that the linguistic properties of a speaker’s native language affect the cognitive processing of other languages. |
| Approach: | They found that the correlation between eye movements and native language similarity may be more complex than the original study found. |
| Outcome: | The proposed model shows that the correlation between eye movements and native language similarity may be more complex than the original study. |
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)
Copied to clipboard
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
| Challenge: | Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. |
| Approach: | They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations. |
| Outcome: | The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work. |
Effective Crowdsourcing of Multiple Tasks for Comprehensive Knowledge Extraction (2020.lrec-1)
Copied to clipboard
Sangha Nam, Minho Lee, Donghwan Kim, Kijong Han, Kuntae Kim, Sooji Yoon, Eun-kyung Kim, Key-Sun Choi
| Challenge: | Existing studies on information extraction from unstructured texts lack a coherent evaluation of all tasks. |
| Approach: | They propose to use crowdsourcing data to develop a Korean information extraction initiative point . they propose to train and evaluate four Korean information extracting tasks using a state-of-the-art model . |
| Outcome: | The proposed model will be used to evaluate four Korean information extraction tasks using crowdsourcing data. |
Parallax: Visualizing and Understanding the Semantics of Embedding Spaces via Algebraic Formulae (P19-3)
Copied to clipboard
| Challenge: | Embeddings are a fundamental component of many modern machine learning and natural language processing models. |
| Approach: | They propose a tool for visualizing embedding spaces using parametric projections . they demonstrate the power of Parallax and propose % task-oriented approach . |
| Outcome: | The proposed tool is based on two-dimensional projections without interpretable semantics . it enhances interpretability and allows for more fine-grained analysis . |
Diachronic degradation of language models: Insights from social media (P18-2)
Copied to clipboard
| Challenge: | Existing studies have explored whether and how language models degrade over time, i.e. why they fail to work on contemporary language. |
| Approach: | They investigate the accuracy of pre-trained language models for downstream tasks in machine learning and user profiling. |
| Outcome: | The results show that it is possible to measure diachronic drifts within social media and within the span of a few years. |
DAVE: Differential Diagnostic Analysis Automation and Visualization from Clinical Notes (2023.eacl-demo)
Copied to clipboard
| Challenge: | a tool that uses natural language processing and machine learning to help visualize diagnostic algorithms in real-time. |
| Approach: | They propose a tool that uses natural language processing and machine learning to visualize diagnostic algorithms in real-time. |
| Outcome: | The proposed system automates the selection and visualization process of diagnostic algorithms. |
Sentiment Analysis of Yelp Review Dataset: A Comparative Study of Machine Learning Methods (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for sentiment analysis are inconsistent and require manual processing. |
| Approach: | They use natural language processing and machine learning to classify Yelp reviews' sentiments. |
| Outcome: | The proposed model outperforms other models on Yelp reviews. |
Alfred: A System for Prompted Weak Supervision (2023.acl-demo)
Copied to clipboard
| Challenge: | Alfred is the first system for programmatic weak supervision (PWS) that creates training data for machine learning by prompting. |
| Approach: | They propose to use Python to create training data by prompting for machine learning . they find that it improves query throughput by 2.9x versus a naive approach . |
| Outcome: | The proposed system improves query throughput by 2.9x versus a naive approach. |
Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing databases contain tens of millions of molecules; PubChem alone has 110 million compounds. |
| Approach: | They propose a task to retrieve molecules using natural language descriptions as queries . they construct a paired dataset of molecules and their corresponding text descriptions . |
| Outcome: | The proposed approach improves results from 0.372 to 0.499 MRR. |
Vote’n’Rank: Revision of Benchmarking with Social Choice Theory (2023.eacl-main)
Copied to clipboard
Mark Rofin, Vladislav Mikhailov, Mikhail Florinsky, Andrey Kravchenko, Tatiana Shavrina, Elena Tutubalina, Daniel Karabekyan, Ekaterina Artemova
| Challenge: | ML benchmarks have been criticized for their construct validity, fragility of the design and task choices. |
| Approach: | They propose a framework for ranking systems in multi-task benchmarks under the principles of the social choice theory and propose 'vote'n'rank' procedures are more robust than the mean average while being able to handle missing performance scores and determine conditions under which the system becomes the winner. |
| Outcome: | The proposed framework can be utilised to draw new insights on benchmarking in several ML sub-fields and identify the best-performing systems in research and development case studies. |
Meta Learning for Natural Language Processing: A Survey (2022.naacl-main)
Copied to clipboard
| Challenge: | Meta-learning is an emerging field in machine learning, but there is no systematic survey of these approaches in NLP. |
| Approach: | They propose to introduce meta-learning and the common approaches and summarize their work and review their work in the NLP community. |
| Outcome: | The proposed methods improve performance in many NLP tasks but are limited to domains, languages, countries, or styles. |
SRL4ORL: Improving Opinion Role Labeling Using Multi-Task Learning with Semantic Role Labeling (N18-1)
Copied to clipboard
| Challenge: | Recent neural approaches do not outperform the state-of-the-art feature-based models for Opinion Role Labeling (ORL). |
| Approach: | They propose to use multi-task learning to improve Opinion Role Labeling by using a related task which has substantially more data. |
| Outcome: | The proposed model outperforms the state-of-the-art model for Opinion Role Labeling (ORL) with more data. |
Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case (2020.coling-main)
Copied to clipboard
| Challenge: | Semantic embeddings have advanced the state of the art for natural language processing tasks . but their inner workings are poorly understood and there is a shortage of analysis tools . |
| Approach: | They propose to extend visual-semantic embeddings to multimodal domains by defining probing tasks for embeddable image-caption pairs and testing them with classifiers. |
| Outcome: | The proposed probing tasks show up to 16% more accurate on visual-semantic embeddings compared to unimodal embedders . the proposed extensions to multimodal domains have been lauded as promising in natural language processing . |
Neural Naturalist: Generating Fine-Grained Image Comparisons (D19-1)
Copied to clipboard
| Challenge: | a dataset of 41k sentences describes fine-grained differences between photographs of birds . human observers are adept at making fine-grain comparisons, but sometimes require aid in distinguishing visually similar classes. |
| Approach: | They propose a model that generates comparative language from a dataset of 41k sentences describing fine-grained differences between photographs of birds. |
| Outcome: | The proposed model can explain differences in visual embedding space using natural language . it evaluates the results with humans who must use the descriptions to distinguish real images . |
AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework (2024.lrec-main)
Copied to clipboard
| Challenge: | Currently, ML&DL methods fail to provide reasons for stock trend predictions, lacking interpretability and reasoning processes. large language models (LLMs) suffer from hallucinations and are unable to keep up with the latest information. |
| Approach: | They develop a method to train large language models to handle financial analysis tasks . they use AlphaFin datasets to compare performance with traditional methods . |
| Outcome: | The proposed method improves stock trend prediction and financial question answering tasks. |
Harnessing LLMs for Temporal Data - A Study on Explainable Financial Time Series Forecasting (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in machine learning and artificial intelligence have opened up numerous opportunities and challenges in financial time series forecasting. |
| Approach: | They propose to use Large Language Models for explainable financial time series forecasting to leverage cross-sequence information and extract insights from text and price time series. |
| Outcome: | The proposed model outperforms ARMA-GARCH and gradient-boosting tree models while underperforming on other models. |
Crowdsourced Corpus of Sentence Simplification with Core Vocabulary (L18-1)
Copied to clipboard
| Challenge: | a crowdsourced corpus of simplified sentences is used to generate complex sentences from more complex ones. |
| Approach: | They propose to use crowdsourced data set of simplified sentences from Japanese textbooks and reference books to generate simplified sentences. |
| Outcome: | The proposed set of simplified sentences is a good quality data set for machine learning. |
Autism Detection in Speech – A Survey (2024.findings-eacl)
Copied to clipboard
| Challenge: | a range of studies have been done on autism in voice, speech and language . females are under-researched in the field, and there are few experiments with transformers . |
| Approach: | They analyse studies of how autism is displayed in voice, speech and language . they define autism and which comorbidities might influence the correct detection . |
| Outcome: | The authors show that there is already a lot of research on autism in speech, but there are still some shortcomings. |
Annotating Spin in Biomedical Scientific Publications : the case of Random Controlled Trials (RCTs) (L18-1)
Copied to clipboard
| Challenge: | Fig. 6: Annotation of biomedical abstracts for automatic detection of inadequate claims (spin) spin is a misleading presentation of scientific results in randomized controlled trials, an important type of clinical trial. |
| Approach: | They propose an algorithm for automatic detection of inadequate claims (spin) they propose to use a corpus of biomedical articles for the task . |
| Outcome: | The proposed algorithm can detect inadequate claims in biomedical abstracts without requiring any prior knowledge of the literature. |
Answerable or Not: Devising a Dataset for Extending Machine Reading Comprehension (C18-1)
Copied to clipboard
| Challenge: | Existing MRC algorithms assume that each question is answerable by looking at text passages, but to realize human-like language comprehension ability, a machine should be able to distinguish not-answerable questions from answerable questions. |
| Approach: | They propose a method for automatically assigning difficulty level labels to a dataset that alters an existing MRC dataset and describes the resulting dataset. |
| Outcome: | The proposed method can detect NAQs in a dataset with difficulty level labels and is valid and potentially useful in the development of advanced MRC models. |
What Action Causes This? Towards Naive Physical Action-Effect Prediction (P18-1)
Copied to clipboard
| Challenge: | a new task on naive physical action-effect prediction addresses the relationship between concrete actions and their effects on the state of the physical world as depicted by images. |
| Approach: | They propose a task that harnesses web image data to facilitate action-effect prediction. |
| Outcome: | The proposed approach harnesses web image data through distant supervision to facilitate learning for action-effect prediction. |
Automated Scoring: Beyond Natural Language Processing (C18-1)
Copied to clipboard
| Challenge: | In this paper, we argue that building operational automated scoring systems is a task that has disciplinary complexity above and beyond competitive shared tasks. |
| Approach: | They argue that building operational automated scoring systems is a task that has disciplinary complexity above and beyond standard competitive shared tasks . they argue that it is essential for us as NLP researchers to understand and incorporate these perspectives in our research and work towards a mutually satisfactory solution . |
| Outcome: | The proposed approach is based on the findings of a recent conference on automated scoring. |
SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have transformed machine learning but have raised significant legal concerns due to their potential to produce text that infringes on copyrights. |
| Approach: | They propose a lightweight, real-time defense mechanism to prevent the generation of copyrighted text by evaluating methods and testing attack strategies. |
| Outcome: | The proposed defense significantly reduces the volume of copyrighted text generated by LLMs by effectively refusing malicious requests. |
Extracted BERT Model Leaks More Information than You Think! (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models are vulnerable to model extraction attacks . model extraction can cause severe privacy leakage even when victim models are facilitated with state-of-the-art defensive strategies. |
| Approach: | They propose to launch an attribute-inference attack against an extracted BERT model to prevent privacy leakage. |
| Outcome: | The proposed attack can cause severe privacy leakage even when victim models are facilitated with state-of-the-art defensive strategies. |
Corpus REDEWIEDERGABE (2020.lrec-1)
Copied to clipboard
| Challenge: | The corpus REDEWIEDERGABE contains detailed annotations for speech, thought and writing representation (ST&WR) with approximately 490,000 tokens, it is the largest resource of its kind. |
| Approach: | This paper presents corpus REDEWIEDERGABE, a German-language historical corpus with detailed annotations for speech, thought and writing representation (ST&WR). |
| Outcome: | The corpus REDEWIEDERGABE contains 490,000 tokens and is the largest resource of its kind. |
Nuanced Toxicity Detection in Spanish: A New Corpus and Benchmark Study (2026.findings-eacl)
Copied to clipboard
Alba María Mármol-Romero, Robiert Sepúlveda-Torres, Estela Saquete, María-Teresa Martín-Valdivia, L. Alfonso Ureña
| Challenge: | Existing corpora for Spanish are under-resourced for toxic content detection . sarcasm, indirect aggression, irony, and other toxicity are not detected in English . |
| Approach: | They propose to extend the NECOS-TOX corpus to include 4,011 Spanish comments . each comment is annotated across three levels of toxicity, with substantial inter-annotator agreement . |
| Outcome: | The proposed model performs on par with larger models and is released publicly . the proposed model is based on a human-in-the-loop active learning strategy . |
Rogue Scores (2023.acl-long)
Copied to clipboard
| Challenge: | Critical evaluation decisions and parameters are routinely omitted, making most reports irreproducible . Thousands of papers use nonstandard evaluation packages with software defects that produce incorrect scores. |
| Approach: | a systematic review of over two thousand papers using a popular metric called ROUGE finds errors . critical evaluation decisions and parameters are routinely omitted, making most reported scores irreproducible . a large number of ROUGEE model evaluation scores have been incorrectly computed . |
| Outcome: | a systematic review of over two thousand papers finds that ROUGE scores are incorrect . the metric is widely used in machine learning and is inconsistent with human evaluations . |
Commonsense Knowledge Mining from Pretrained Models (D19-1)
Copied to clipboard
| Challenge: | Several approaches have been proposed for training models for commonsense knowledge base completion (CKBC) due to the sparsity of training data. |
| Approach: | They propose a method for generating commonsense knowledge using a large, pre-trained bidirectional language model by transforming relational triples into masked sentences. |
| Outcome: | The proposed method outperforms models trained on held-out test sets on a held-up set, suggesting that it generalizes better than current supervised methods. |
TextAnnotator: A UIMA Based Tool for the Simultaneous and Collaborative Annotation of Texts (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing annotation tools are not efficient for the annotation of corpora and are not error-free. |
| Approach: | They propose to extend existing annotation tools by evaluating their flexibility and efficiency. |
| Outcome: | The proposed system performs platform-independent multimodal annotations and annotates complex textual structures. |
GCRC: A New Challenging MRC Dataset from Gaokao Chinese for Explainable Evaluation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing machine reading comprehension datasets lack an explainable evaluation of systems' reasoning capabilities. |
| Approach: | They propose a dataset with multi-choice questions that evaluates MRC systems' reasoning process . they use sentence-level relevant supporting facts, error reason of distractors to evaluate MRC . |
| Outcome: | The proposed dataset is more challenging and useful for identifying limitations of existing MRC systems in an explainable way. |
Polarized Opinion Detection Improves the Detection of Toxic Language (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for estimating polarized annotations are un-normalized and difficult to exploit in machine learning. |
| Approach: | They propose a method for K-class text classification that exploits polarized texts in the dataset. |
| Outcome: | The proposed method exploits polarized texts in a dataset and can improve classification performance. |
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021.eacl-main)
Copied to clipboard
| Challenge: | et al., 2018a): a poor phrasing may make the conversation go awry. |
| Approach: | They propose a model that can help suggest rephrasings of toxic comments in a more civil manner. |
| Outcome: | The proposed model generates sentences that are more fluent and better at preserving the initial content compared to earlier systems and human evaluation. |
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels (2024.naacl-long)
Copied to clipboard
| Challenge: | a line of recent work has illustrated that annotators disagree for many reasons . capturing disagreements can improve model performance and calibration, authors argue . |
| Approach: | They propose a new paradigm shift in data labeling for machine learning that challenges annotator disagreement by treating disagreement as a valuable source of information. |
| Outcome: | The proposed approaches challenge annotator disagreement and provide recommendations for the data labeling pipeline and avenues for future research. |
Adversarial Domain Adaptation for Duplicate Question Detection (D18-1)
Copied to clipboard
| Challenge: | Recent years have seen the rise of community question answering forums . duplicate questions easily become ubiquitous as users often ask the same question, possibly in a slightly different formulation, making it difficult to find the best (or one correct) answer. |
| Approach: | They propose to use domain adaptation to detect duplicate questions in forums . they find that domain adaptation improves performance over multiple pairs of domains . |
| Outcome: | The proposed approach improves 5.6% over the best baseline across multiple pairs of domains. |
Research Replication Prediction Using Weakly Supervised Learning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to predict scientific claims’ replicability use only hand-extracted statistics features without utilizing research papers’ text information. |
| Approach: | They propose two weakly supervised learning approaches that use automatically extracted text information of research papers to improve the prediction accuracy of research replication using both labeled and unlabeled datasets. |
| Outcome: | The proposed methods achieve an accuracy of 75.76% over real-world datasets. |
Avoiding Overlap in Data Augmentation for AMR-to-Text Generation (2021.acl-short)
Copied to clipboard
| Challenge: | Using unlabeled data to boost model performance is common practice in machine learning and natural language processing. |
| Approach: | They propose methods for excluding parts of Gigaword to remove overlap . they propose to use the AMR dataset for AMR-to-text generation . |
| Outcome: | The proposed approach leads to a more realistic evaluation of the task of AMR-to-text generation. |
A Unifying View On Task-oriented Dialogue Annotation (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent research attention in task-oriented dialogue systems focuses on end-to-end neural models. |
| Approach: | They present a dataset that combines annotated corpora from four domains to provide a unified ontology and annotation schema for task-oriented dialogues. |
| Outcome: | The proposed dataset improves language, information content and performance in dialogues with two recent models. |
Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies show that the energy requirements of current NLP models are growing at a rapid, unsustainable pace. |
| Approach: | They investigate ways to measure energy usage and different hardware settings that can be tuned to reduce energy consumption for training and inference for language models. |
| Outcome: | The proposed techniques can reduce energy consumption for training and inference for language models. |
Towards Lossless Encoding of Sentences (P19-1)
Copied to clipboard
| Challenge: | Existing methods for encoding text into lossless representations focus on performing well on downstream tasks and are unable to reconstruct original sequence from learned embedding. |
| Approach: | They propose a lossless method for encoding long sequences of texts into feature rich representations by recursive autoencoding. |
| Outcome: | The proposed method performs well on sentiment analysis and sentiment classification tasks. |
Reconstruction Attack on Instance Encoding for Language Understanding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing private learning schemes which protect data privacy can be used to train models using instance encoding. |
| Approach: | They propose to recover the private training data and use it to break a private learning scheme TextHide. |
| Outcome: | The proposed attack would advance privacy-preserving machine learning in the context of natural language processing. |
Through the Looking Glass: Learning to Attribute Synthetic Text Generated by Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have enabled synthetic text generation that is often comparable to the organic text. |
| Approach: | They propose and test several ML-based methods to attribute authorship of synthetic text to language models (LMs) they propose to use a fine-tuned version of XLNet to achieve excellent accuracy . |
| Outcome: | The proposed method achieves excellent accuracy (91% to near perfect 98%) across a range of experiments where the synthetic text may be generated using pre-trained LMs, fine-tuned LM, or by varying text generation parameters. |
On the Empirical Complexity of Reasoning and Planning in LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Evidence shows that the relative performance of CoT, ToT, and their variants may vary from task to task. |
| Approach: | They propose to use chain-of-thought (CoT), tree-of thought (ToT), and related techniques to solve complex reasoning tasks with Large Language Models. |
| Outcome: | The proposed methods outperform the linear structure of CoT on hard reasoning tasks. |
Habibi - a multi Dialect multi National Arabic Song Lyrics Corpus (2020.lrec-1)
Copied to clipboard
| Challenge: | Unlike western music, Arabic songs are poorly classified and the majority of the songs available online are classified under Modern Arabic Pop genre or what is now known as Franco-Arabic . |
| Approach: | They introduce Habibi the first Arabic Song Lyrics corpus for singers from 18 different Arabic countries. |
| Outcome: | The proposed corpus contains more than 30,000 Arabic song lyrics in 6 Arabic dialects for singers from 18 different arab countries. |
The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild (2022.lrec-1)
Copied to clipboard
| Challenge: | GINCO is a new training dataset for automatic genre identification based on 1,125 crawled Slovenian web documents that consist of 650,000 words. |
| Approach: | They propose to use 1,125 crawled Slovenian web documents to train a new genre classification system based on a GINCO training dataset . |
| Outcome: | The proposed classifiers perform better on the 1,125 crawled Slovenian web documents than the existing models and achieve higher scores on the task. |
Linguistic Cues to Deception and Perceived Deception in Interview Dialogues (N18-1)
Copied to clipboard
| Challenge: | a recent study examined deception detection in several domains, including fake reviews, mock crime scenes, and opinions about topics such as abortion or the death penalty. |
| Approach: | They analyze linguistic features in truthful and deceptive interview dialogues . they also examine interviewer perceptions of deception, identifying characteristics of deceptives . |
| Outcome: | The proposed model outperforms human classifications using linguistic features and individual traits. |
MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to automating ML are time-consuming and difficult to understand for human developers. |
| Approach: | They propose a framework that leverages large language models to develop ML solutions for novel tasks. |
| Outcome: | The proposed framework bridges the gap between machine intelligence and human knowledge by exploiting state-of-the-art large language models. |
Annotating and Analyzing Biased Sentences in News Articles using Crowdsourcing (2020.lrec-1)
Copied to clipboard
| Challenge: | a lack of publicly available news bias datasets has hindered efforts to detect subtle biases in news articles. |
| Approach: | They propose a news bias dataset which contains sentences with bias labels . they propose to use the dataset to develop and evaluate methods for detecting news bias . |
| Outcome: | The proposed dataset can be used for analyzing news bias and for developing and evaluating methods for news bias detection. |
Understanding “Democratization” in NLP and ML Research (2024.emnlp-main)
Copied to clipboard
| Challenge: | a large number of NLP and ML papers mention terms related to democracy . authors find that democratization is most frequently used to convey (ease of) access to or use of technologies without meaningfully engaging with theories of democratisation. |
| Approach: | They analyze papers using the term "democra*" to clarify how it is understood in NLP and ML . they find that democratization is most frequently used to convey (ease of) access to or use of technologies . |
| Outcome: | The authors analyze papers using the term "democra*" they find that democratization is most frequently used to convey (ease of) access to or use of technologies without meaningfully engaging with theories of democratisation. |
S2abEL: A Dataset for Entity Linking from Scientific Tables (2023.emnlp-main)
Copied to clipboard
| Challenge: | Entity linking (EL) is a longstanding problem in natural language processing and information extraction. |
| Approach: | They propose a neural baseline method for EL on scientific tables containing many out-of-knowledge-base mentions and a method that significantly outperforms a generic table EL method. |
| Outcome: | The proposed method significantly outperforms state-of-the-art generic table EL method on scientific tables with many out-of knowledge-base mentions. |
A Pragmatic Approach for Classical Chinese Word Segmentation (L18-1)
Copied to clipboard
| Challenge: | Classical Chinese word segmentation is largely neglected due to its obsoleteness . a new approach to segmentation using a marked-up corpus is needed . |
| Approach: | They propose a pragmatic approach to deal with Classical Chinese word segmentation without any marked-up corpus. |
| Outcome: | The proposed method makes the CCWS without any marked-up corpus more accurate compared with collocation-based methods. |
Text Classification Under Class Distribution Shift: A Survey (2026.eacl-long)
Copied to clipboard
| Challenge: | ML models assume that training and test data are sampled from the same distribution, but in daily practice, this assumption is often broken. |
| Approach: | They survey articles studying open-set text classification to understand the distribution shifts and mitigation approaches for each problem setup. |
| Outcome: | The proposed methods can solve problems caused by the shifting class distribution in open-set text classification and related tasks. |
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives (2024.naacl-long)
Copied to clipboard
Vinodkumar Prabhakaran, Christopher Homan, Lora Aroyo, Aida Mostafazadeh Davani, Alicia Parrish, Alex Taylor, Mark Diaz, Ding Wang, Gregory Serapio-García
| Challenge: | Recent work shows that ignoring rater subjectivity is problematic within specific tasks and for specific subgroups. |
| Approach: | They propose a disagreement analysis framework to measure group association in perspectives among different rater subgroups. |
| Outcome: | The proposed framework reveals specific rater groups that have significantly different perspectives than others on certain tasks and helps identify demographic axes that are crucial to consider in specific task contexts. |
Reproducibility Issues for BERT-based Evaluation Metrics (2022.emnlp-main)
Copied to clipboard
| Challenge: | Reproducibility is of utmost concern in machine learning and natural language processing . lexical-overlap metrics are still the dominant metric in natural language generation . |
| Approach: | They ask whether results and claims from four recent BERT-based evaluation metrics can be reproduced. |
| Outcome: | The proposed metrics outperform the dominant metric, BLEU, and show that they can be reproduced. |
MetaLead: A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing leaderboards capture only the best results from each paper and have limited metadata. |
| Approach: | They propose to create a fully human-annotated ML Leaderboard dataset that captures all experimental results and contains extra metadata. |
| Outcome: | The MetaLead dataset captures all experimental results and contains extra metadata for cross-domain evaluation. |
Empowering Language Understanding with Counterfactual Reasoning (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for language understanding use the recognized patterns in the testing phase that are inherently different from us humans who have counterfactual thinking. |
| Approach: | They propose a counterfactual Reasoning Model which mimics counterfactive thinking by learning from few counterffact samples. |
| Outcome: | The proposed model can detect and make predictions from textual patterns . it can also detect negative sarcastic puns by comparing them with imaginations . |
“My life is miserable, have to sign 500 autographs everyday”: Exposing Humblebragging, the Brags in Disguise (2025.findings-acl)
Copied to clipboard
| Challenge: | Humblebragging is a phenomenon in which individuals present self-promotional statements under the guise of modesty or complaints. |
| Approach: | They propose a task of automatically detecting humblebragging in text and propose '4-tuple definition' they also propose machine learning, deep learning, and large language models to perform the task . |
| Outcome: | The proposed model achieves an F1-score of 0.88 and is non-trivial even for humans. |
Semi-Automatic Construction and Refinement of an Annotated Corpus for a Deep Learning Framework for Emotion Classification (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for emotion classification are expensive and require a large corpus of data. |
| Approach: | They propose a method for creating a semi-automatically constructed emotion corpus by correcting errors in the corpus. |
| Outcome: | The proposed method improves the quality of the emotion labels by correcting errors. |
Automatic Detection of Machine Generated Text: A Critical Survey (2020.coling-main)
Copied to clipboard
| Challenge: | Current text generative models excel in producing text that matches the style of human language reasonably well. |
| Approach: | They conduct an in-depth error analysis of the state-of-the-art detector and discuss research directions to guide future work in this exciting area. |
| Outcome: | The proposed detectors can distinguish between human and text generated by the model and can be used to generate fake news and fake product reviews. |
WIND: Weighting Instances Differentially for Model-Agnostic Domain Adaptation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for instance weighting cannot learn the weights which make the model generalize well in target domain. |
| Approach: | They propose a modelagnostic instance weighting algorithm which can learn the instance weights instead of manually designed weighting metrics. |
| Outcome: | The proposed method can learn the instance weights instead of manually designed weighting metrics. |
KenMeSH: Knowledge-enhanced End-to-end Biomedical Text Labelling (2022.acl-long)
Copied to clipboard
| Challenge: | Medical Subject Headings (MeSH) are manually assigned to every biomedical article to facilitate retrieval of relevant information. |
| Approach: | They propose a model that combines new text features with a dynamic knowledge-enhanced mask attention that integrates document features with MeSH label hierarchy and journal correlation features to index MeSH terms. |
| Outcome: | The proposed model achieves state-of-the-art on a number of measures. |
Improving homograph disambiguation with supervised machine learning (L18-1)
Copied to clipboard
| Challenge: | a new system for text-to-speech synthesis uses rule-based homograph disambiguation . a simple application of machine learning produces significant improvements in homograph ambiguity . |
| Approach: | They propose a rule-based homograph disambiguation system for text-to-speech synthesis at Google . they compare it to a new system which performs disambiguations using classifiers trained on labeled data . |
| Outcome: | The proposed system is more accurate than hand-written rules or machine learning alone. |
A Study on the Ambiguity in Human Annotation of German Oral History Interviews for Perceived Emotion Recognition and Sentiment Analysis (2022.lrec-1)
Copied to clipboard
Michael Gref, Nike Matthiesen, Sreenivasa Hikkal Venugopala, Shalaka Satheesh, Aswinkumar Vijayananth, Duc Bach Ha, Sven Behnke, Joachim Köhler
| Challenge: | Sentiment analysis and emotion recognition can help research in audiovisual interview archives . however, humans perceive sentiments and emotions ambiguously and subjectively . |
| Approach: | They investigate human perceptions of emotions and sentiments in oral history interviews . they show that human perception for different emotions is ambiguous and subjective . authors propose deep learning as a way to categorize and search emotions . |
| Outcome: | The proposed techniques can be used to search and index audiovisual interviews . the authors show that human perceptions differ for different emotions . |
Argument Mining for Understanding Peer Reviews (N19-1)
Copied to clipboard
| Challenge: | In 2015 alone, approximately 63.4 million hours were spent on peer reviews. |
| Approach: | They propose to automatically detect argumentative propositions put forward by reviewers and their types by automatically detecting their types and types. |
| Outcome: | The proposed method detects (1) the argumentative propositions put forward by reviewers, and (2) their types (e.g., evaluating the work or making suggestions for improvement). |
On the Computational Modelling of Michif Verbal Morphology (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing computational models of the verbal morphology of the Métis language are insufficient to model the language's unique phonological interactions. |
| Approach: | They propose a finite-state computational model of the verbal morphology of Michif . they use composed finite state transducers to model concatenative morphologies . |
| Outcome: | The proposed model is based on a series of finite-state transducers. |
Evaluation of Lifelong Learning Systems (2020.lrec-1)
Copied to clipboard
| Challenge: | Current intelligent systems need the expensive support of machine learning experts to sustain their performance level when used on a daily basis. |
| Approach: | They propose a generic evaluation methodology for lifelong learning systems . they use "initialisation data" to refer to the set of training, development and test data together . |
| Outcome: | The proposed evaluation method is based on the evaluation of human-assisted learning outside the context of lifelong learning. |
Annotation of a Large Clinical Entity Corpus (D18-1)
Copied to clipboard
| Challenge: | Past researches have shown the superiority of statistical/ML approaches over the rule based approaches. |
| Approach: | They propose to annotate a clinical domain annotated corpus using a small data set or a narrower domain to take full advantage of machine learning. |
| Outcome: | The proposed corpus contains 5,160 clinical documents from forty different clinical specialties. |
Building Question-Answer Data Using Web Register Identification (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in web register (genre) identification have created a shortage of QA datasets for English and Finnish. |
| Approach: | They propose a machine learning-based method for extracting QA pairs from web-scale data using XLM-R and a multilingual CORE web register corpus . they then develop a NER-style token classifier to identify the QA text spans within these documents. |
| Outcome: | The proposed method is adaptable to any language given the availability of language models and extensive web data, but it is limited to English and Finnish. |
Models Teaching Models: Improving Model Accuracy with Slingshot Learning (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods to label datasets are expensive and require human labor . a semi-supervised method that augments a small dataset with labels reduces the cost of using simpler methods . |
| Approach: | They propose a semi-supervised method to augment a human-labeled dataset with labels from a teacher model to slingshot the performance of a student model. |
| Outcome: | The proposed method reduces the accuracy trade-off required to use simpler methods without disrupting their benefits. |
FeRG-LLM : Feature Engineering by Reason Generation Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | FeRG-LLM is a large language model that performs feature engineering at an 8billion-parameter scale. |
| Approach: | They propose a framework to perform feature engineering at an 8billion-parameter scale using conversational dialogues. |
| Outcome: | The proposed framework outperforms Llama 3.1 70B and Llma 3.2 on most datasets while using fewer resources and achieving reduced inference time. |
A Federated Approach for Hate Speech Detection (2023.eacl-main)
Copied to clipboard
| Challenge: | Despite the scale of social media content, privacy preservation in hate speech detection has remained understudied. |
| Approach: | They propose to use federated machine learning to address privacy concerns in hate speech detection by obtaining a 6.81% improvement in F1-score. |
| Outcome: | The proposed method improves the F1-score of hate speech detection by 6.81% while maintaining public data privacy. |
An Annotated Corpus of Textual Explanations for Clinical Decision Support (2022.lrec-1)
Copied to clipboard
Roland Roller, Aljoscha Burchardt, Nils Feldhus, Laura Seiffe, Klemens Budde, Simon Ronicke, Bilgin Osmanodja
| Challenge: | In recent years, machine learning for clinical decision support has gained more and more attention. |
| Approach: | They propose to use XAI to provide an explanation of a model's decision making process by constructing a corpus of sentences that are annotated with different semantic layers. |
| Outcome: | The proposed models outperform physicians on very specific, narrow tasks or can help physicians to work more efficiently. |
Recognition of Implicit Geographic Movement in Text (2020.lrec-1)
Copied to clipboard
| Challenge: | a growing field of research is analyzing the geographic movement of humans, animals, and other entities. |
| Approach: | They created a corpus of sentences labeled as describing geographic movement or not . they used hand labeling, crowd voting and machine learning to predict more labels . |
| Outcome: | a new method uses hand labeling, crowd voting and machine learning to predict more labels. |
Comparing Machine Learning and Deep Learning Approaches on NLP Tasks for the Italian Language (2020.lrec-1)
Copied to clipboard
| Challenge: | Using available datasets, we compare deep learning and traditional machine learning methods for various NLP tasks in Italian. |
| Approach: | They compare deep learning and traditional machine learning methods for various NLP tasks in Italian. |
| Outcome: | The proposed methods outperform traditional methods in sequence tagging tasks and classification tasks in Italian. |
A Diagnostic Study of Explainability Techniques for Text Classification (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing explainability techniques that can be produced post-hoc with already trained models are lacking a definitive guide on how to choose one given a particular task and model architecture. |
| Approach: | They propose to use a list of diagnostic properties to evaluate existing explainability techniques to compare them with human annotations of salient input regions. |
| Outcome: | The proposed list compares a set of explainability techniques on downstream text classification tasks and neural network architectures. |
Challenges with Sign Language Datasets for Sign Language Recognition and Translation (2022.lrec-1)
Copied to clipboard
Mirella De Sisto, Vincent Vandeghinste, Santiago Egea Gómez, Mathieu De Coster, Dimitar Shterionov, Horacio Saggion
| Challenge: | Sign Languages are the primary means of communication for at least half a million people in Europe . however, the development of SL recognition and translation tools is slowed down by resource scarcity and data formats are not suitable for machine learning. |
| Approach: | They propose a framework to unify available resources and facilitate SL research for different languages. |
| Outcome: | The proposed framework is based on a set of ELAN files and returns textual and visual data ready to train SL recognition and translation models. |
EmpathBERT: A BERT-based Framework for Demographic-aware Empathy Prediction (2021.eacl-main)
Copied to clipboard
| Challenge: | EmpathBERT is a demographic-aware framework for empathy prediction based on BERT. |
| Approach: | They propose a demographic-aware framework for empathy prediction based on BERT and utilize user demographics to analyze user responses to stimulative news articles. |
| Outcome: | The proposed framework surpasses machine learning and deep learning models and highlights the importance of demographic information in the responses. |
Is Attention Explanation? An Introduction to the Debate (2022.acl-long)
Copied to clipboard
Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang, Thomas François, Patrick Watrin
| Challenge: | Attention has been used in various tasks of NLP and other fields of machine learning to increase performance and provide some explanations. |
| Approach: | They propose to use attention as an explanation for deep learning models to increase performance . they propose to apply attention weights to queries and queries based on scalar scores . |
| Outcome: | The proposed model can be used to increase performance while providing some explanations. |
How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social Impact (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent years have seen many breakthroughs in natural language processing (NLP), transitioning it from a mostly theoretical field to one with many real-world applications. |
| Approach: | They propose a moral philosophy definition of social good and a framework to evaluate the direct and indirect real-world impact of NLP tasks. |
| Outcome: | The proposed framework evaluates the direct and indirect real-world impact of NLP tasks and adopts the methodology of global priorities research to identify priority causes for NLP research. |
Automated Fact Checking: Task Formulations, Methods and Future Directions (C18-1)
Copied to clipboard
| Challenge: | Recent research on fact checking has focused on misinformation . however, relevant papers and articles have been published in research communities that are unaware of each other and use inconsistent terminology. |
| Approach: | They propose avenues for future NLP research on automated fact checking . they highlight the use of evidence as an important distinguishing factor . |
| Outcome: | The proposed methods unify the task formulations and methodologies across papers and authors. |
Deconvolutional Time Series Regression: A Technique for Modeling Temporally Diffuse Effects (D18-1)
Copied to clipboard
| Challenge: | a confound exists in time series data that violates assumptions of linear models . time series may violate assumptions through temporal diffusion . |
| Approach: | They propose a statistical model that borrows from digital signal processing to fit latent impulse response functions of arbitrary shape. |
| Outcome: | The proposed model recovers true latent IRFs and improves prediction quality . it is based on a new technique that borrows from digital signal processing . |
Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing (2021.naacl-main)
Copied to clipboard
| Challenge: | Ethical considerations regarding the use of crowdworkers are limited to labor conditions . the Final Rule did not anticipate the use online crowdsourcing platforms for data collection . |
| Approach: | They propose to reopen discussion regarding ethical use of crowdworkers in NLP research . they propose to use online crowdsourcing platforms to evaluate risk of harm . |
| Outcome: | The proposed study identifies common scenarios where crowdworkers performing NLP tasks are at risk of harm. |
Effective Data Augmentation for Sentence Classification Using One VAE per Class (2022.coling-1)
Copied to clipboard
| Challenge: | Variational auto-encoders and its conditional variant the Conditional-VAE (CVAE) are often used to generate new textual data, but they require more complex manipulations to ensure that the generated examples are useful. |
| Approach: | They propose a simple way to use Variational Auto-Encoders (VAE) for data augmentation by training one VAE per class. |
| Outcome: | The proposed method outperforms generative models on binary classification tasks and several dataset sizes on four different tasks. |
Word Segmentation by Separation Inference for East Asian Languages (2022.findings-acl)
Copied to clipboard
| Challenge: | Chinese Word Segmentation (CWS) is a sequence labeling task that divides sentences into words . despite diverse tagging schemas, they all carry implicit position information. |
| Approach: | They propose to model the separation state of every two consecutive characters by tagging them as two tags. |
| Outcome: | The proposed framework outperforms state-of-the-art on Japanese and Korean Word Segmentation datasets. |
When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications? (2024.findings-emnlp)
Copied to clipboard
Yanjun Gao, Skatje Myers, Shan Chen, Dmitriy Dligach, Timothy Miller, Danielle Bitterman, Matthew Churpek, Majid Afshar
| Challenge: | Numerical data is pivotal for medical questions and answers, but tabular data is not fully integrated into LLMs. |
| Approach: | They examine the effectiveness of vector representations from last hidden states of LLMs for medical diagnostics and prognostics using electronic health record data. |
| Outcome: | The proposed representations outperform those using raw numerical EHR data in medical diagnostics and prognostics. |
Lexical and Semantic Features for Cross-lingual Text Reuse Classification: an Experiment in English and Latin Paraphrases (L18-1)
Copied to clipboard
| Challenge: | Analyzing historical languages is challenging because they lack primary material for certain time periods . under-resourced languages such as Ancient Greek and Latin lack advanced natural-language processing (NLP) techniques . |
| Approach: | They propose to use machine learning to detect and classify paraphrastic text reuse in historical texts. |
| Outcome: | The proposed method improves the accuracy of paraphrastic text reuse detection in historical languages. |
Distill and Replay for Continual Language Learning (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models fail to isolate acquired knowledge and forget previously learned tasks when learning in a stream where data distribution may shift. |
| Approach: | They propose a framework that distills knowledge and replays experience from previous tasks when fitting on a new task. |
| Outcome: | The proposed framework outperforms state-of-the-art models in continuously learning tasks of the same type but from different domains, as well as tasks of different types. |
On the Multilingual Capabilities of Very Large-Scale English Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Generative Pre-trained Transformers (GPTs) have been scaled to unprecedented sizes in the history of machine learning. |
| Approach: | They investigate the potential and limits of Generative Pre-trained Transformers in three tasks . they find it can be almost as useful for many languages as it is for English . |
| Outcome: | The proposed model can perform tasks in five different languages, and its potential is explored . it can learn from a few examples "via text interaction" and is scalable to many languages . |
Resource of Wikipedias in 31 Languages Categorized into Fine-Grained Named Entities (2022.coling-1)
Copied to clipboard
| Challenge: | a resource of Wikipedias in 31 languages is categorized into Extended Named Entity (ENE) ENE version 8 has 219 fine-grained NE categories. |
| Approach: | They describe a resource of Wikipedias in 31 languages categorized into Extended Named Entity (ENE) they first categorized 920 K Japanese Wikipedia pages using machine learning, then shared a task of Wikipedia categorization into 30 languages . |
| Outcome: | The proposed system is based on a dataset of Japanese Wikipedia pages . the dataset shows the best performance among the 30 languages . |
Reservoir Transformers (2021.acl-long)
Copied to clipboard
| Challenge: | Using random initialization, we show that some transformers obtain impressive performance even when some of the layers are frozen. |
| Approach: | They propose to freeze transformer layers and use them to improve performance . they find that the transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. |
| Outcome: | The proposed model improves on translation and language modelling tasks even when some layers are frozen. |
Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications (N19-1)
Copied to clipboard
| Challenge: | Existing approaches focus on improving accuracy and overlook other aspects such as robustness and interpretability. |
| Approach: | They propose adversarial modifications for link prediction models that identify influential facts and evaluate their sensitivity to addition of fake facts. |
| Outcome: | The proposed model evaluates the robustness of the model to the addition of fake facts and the interpretability of the models. |
Impact of Evaluation Methodologies on Code Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methodologies for code summarization tasks do not consider timestamps of code and comments. |
| Approach: | They propose a time-segmented evaluation methodology for code summarization that considers timestamps of code and comments during evaluation. |
| Outcome: | The proposed evaluation methodology compares with other evaluation methodologies that have been widely used. |
Can LLMs Find a Needle in a Haystack? A Look at Anomaly Detection Language Modeling (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Anomaly detection (AD) is a problem in machine learning, but it is not always competitive on certain datasets. |
| Approach: | They propose a new approach to Anomaly detection based on large pre-trained language models in three modalities. |
| Outcome: | The proposed model beats baselines on anomaly detection when presented as imbalanced classification problem regardless of the concentration of anomalous samples. |
Exploring Text Recombination for Automatic Narrative Level Detection (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing annotation workflows do not scale well to the annotation of complex narrative phenomena. |
| Approach: | They propose a workflow for narrative level detection that includes operationalization and a model . they propose generating training data synthetically to improve the prediction results . |
| Outcome: | The proposed workflow improves predictions by using training data synthetically. |
Embedding Multimodal Relational Data for Knowledge Base Completion (D18-1)
Copied to clipboard
| Challenge: | Existing approaches focus on a finite set of entities, ignoring the variety of data types used in knowledge bases. |
| Approach: | They propose multimodal knowledge base embeddings that use different neural encoders for observed data and different neural decoders to learn embedded entities and multimodal data. |
| Outcome: | The proposed models outperform existing methods with 5-7% accuracy over existing methods. |
Reconsidering Sentence-Level Sign Language Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Historically, sign language machine translation is framed as a sentence-level task . however, there are known intersentential dependencies that are impossible to resolve in isolation. |
| Approach: | They propose a human baseline for sign language translation that substitutes a person into the machine learning task framing instead of providing the entire document as context. |
| Outcome: | The proposed human baseline for sign language translation shows that deaf signers can only understand key parts of the clip in light of additional discourse-level context. |
Generating Textual Explanations for Machine Learning Models Performance: A Table-to-Text Task (2022.lrec-1)
Copied to clipboard
| Challenge: | Numerical tables are widely used to communicate or report the classification performance of machine learning models with respect to a set of evaluation metrics. |
| Approach: | They propose a task where neural models are trained to generate textual explanations based on the metrics’ scores reported in numerical tables. |
| Outcome: | The proposed model outperforms existing methods and can be used to explain the performance of ML models. |
Towards a Perspectivist Turn in Argument Quality Assessment (2025.naacl-long)
Copied to clipboard
| Challenge: | Argument quality is a key aspect of computational argumentation (CA), but it still exhibits a high degree of subjectivity in perception. |
| Approach: | They propose to use a multi-layered classification to target two aspects of argument quality in a systematic review of NLP datasets. |
| Outcome: | The proposed model improves the quality of annotators and their ability to be used in perspectivist research. |
Learning to Extract Structured Entities Using Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) play a pivotal role in extracting structured information from unstructured text. |
| Approach: | They propose to reformulate the task to be entity-centric, enabling the use of diverse metrics that can provide more insights from various perspectives. |
| Outcome: | The proposed model outperforms baselines and human evaluations on the extracted entities. |
Collocations in Russian Lexicography and Russian Collocations Database (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for collocation extraction cannot be considered perfect, argues a new study. |
| Approach: | They propose to build a database that will include dictionary and statistical collocations in Russian . the database will be based on dictionaries and online systems that describe collocation . |
| Outcome: | The proposed database will include dictionary and statistical collocations in Russian . the results can be useful for machine learning and for other NLP tasks . |
Structuring Radiology Reports: Challenging LLMs with Lightweight Models (2025.emnlp-main)
Copied to clipboard
Johannes Moll, Louisa Fay, Asfandyar Azhar, Sophie Ostmeier, Sergios Gatidis, Tim C. Lueth, Curtis Langlotz, Jean-Benoit Delbrouck
| Challenge: | Radiology reports lack a standardized format, limiting both interpretability and machine learning applications. |
| Approach: | They propose to use lightweight encoder-decoder models for structuring radiology reports . they compare models with eight open-source LLMs with prompting and in-context learning . |
| Outcome: | The proposed models outperform eight open-source LLMs on a human-annotated test set. |
A multi-level multi-label text classification dataset of 19th century Ottoman and Russian literary and critical texts (2024.findings-acl)
Copied to clipboard
| Challenge: | a multi-level, multi-label text classification dataset is used to classify over 3000 documents . authors use a classical bag-of-words (BoW) naive Bayes model and three modern LLMs . |
| Approach: | They propose to apply large language models to a multi-level, multi-label text classification dataset . the dataset features literary and critical texts from 19th-century Ottoman Turkish and Russian . |
| Outcome: | The proposed dataset features literary and critical texts from 19th-century Ottoman Turkish and Russian. |
Towards Language Agnostic Universal Representations (P19-1)
Copied to clipboard
| Challenge: | Current representations in machine learning are language dependent . however, fluent bilingual speakers rarely face trouble translating a task learned in one language to another . |
| Approach: | They propose a method to decouple the language from the problem by learning language agnostic representations. |
| Outcome: | The proposed model achieves similar accuracies in a single language and in another language. |
DeepTC – An Extension of DKPro Text Classification for Fostering Reproducibility of Deep Learning Experiments (L18-1)
Copied to clipboard
| Challenge: | a current state of DKPro TC does not allow integration of deep learning . we integrate Keras, DyNet, and DeepLearning4J as proof-of-concept . |
| Approach: | They propose a deep learning extension for the multi-purpose text classification framework DKPro Text Classification. |
| Outcome: | The proposed extension improves readability and reduces redundant source code. |
DeFaktS: A German Dataset for Fine-Grained Disinformation Detection through Social Media Framing (2024.lrec-main)
Copied to clipboard
| Challenge: | Distinctively curated across various news topics, DeFaktS offers an unparalleled insight into disinformation’s diverse characteristics. |
| Approach: | They propose to annotate every structural component and semantic element of a news piece, eliminating the need for external knowledge sources. |
| Outcome: | The proposed dataset contains 105,855 posts with 20,008 meticulously labeled tweets and eliminates the need for external knowledge sources. |
Revitalization of Indigenous Languages through Pre-processing and Neural Machine Translation: The case of Inuktitut (2020.coling-main)
Copied to clipboard
| Challenge: | Indigenous languages have been considered low-resource and/or endangered . authors propose a method to revitalize the language spoken in northern canada . |
| Approach: | They propose to revitalize the Inuktitut language through pre-processing and neural machine translation . they propose to use this technique to perform morphological analysis and neural translation tasks . |
| Outcome: | The proposed approach improves the Inuktitut language compared to the state-of-the-art . the proposed approach is based on preprocessing and neural machine translation . |
Uncovering the Root of Hate Speech: A Dataset for Identifying Hate Instigating Speech (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a lack of comprehensive datasets specifically annotated for hate instigating speech hinders research . lack of reliable models for hate triggering makes it difficult to apply off-the-shelf models to the problem. |
| Approach: | They propose to use a multilingual dataset to identify hate instigating speech . lack of comprehensive datasets specifically annotated for hate instigators hinders their work . |
| Outcome: | The proposed dataset identifies hate instigating speech across languages . lack of comprehensive datasets makes it difficult to train and evaluate models . |
Multi-Path Transformer is Better: A Case Study on Neural Machine Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Extensive experiments on 12 WMT tasks show that shallower multi-path models can achieve similar or even better performance than the deeper model. |
| Approach: | They propose to use a parameter-efficient multi-path structure to fuse features extracted from different paths to achieve better performance. |
| Outcome: | The proposed model can achieve better performance with the same number of parameters than the deeper model. |
Use of Formal Ethical Reviews in NLP Literature: Historical Trends and Current Practices (2021.findings-acl)
Copied to clipboard
| Challenge: | Ethical aspects of research in language technologies have received much attention recently . do we observe a rise in formal ethical reviews of NLP studies? |
| Approach: | They conduct a qualitative and quantitative analysis of the ethics of NLP research . they compare the ethical reviews of NLAs to those of related disciplines . |
| Outcome: | The results compare the ACL Anthology to other related disciplines in the field . the results show that there is a heightened awareness of ethical issues that was previously lacking . |
When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | a number of studies have questioned assumptions of majority vote aggregated labels. |
| Approach: | They construct a model that predicts individual annotator ratings on potentially offensive text and combines this information with the predicted target group of the text to predict the ratings of target group members. |
| Outcome: | The proposed model raises performance over baseline by 22% and 33% at predicting variance among annotators. |
Depth Aware Hierarchical Replay Continual Learning for Knowledge Based Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | Continual learning models adapt well to the latest data but lose ability to remember past data due to changes in the data source. |
| Approach: | They propose a hierarchical replay framework that allows models to keep a small memory of previous learned data that uses replay. |
| Outcome: | The proposed model outperforms previous continual learning methods in mitigating catastrophic forgetting. |
Predicting Clickbait Strength in Online Social Media (2020.coling-main)
Copied to clipboard
| Challenge: | Clickbaits are sensational, provocative or controversial posts that entice readers to click on them. |
| Approach: | They propose to model clickbait strength prediction using transformers to predict clickbaiting intensity. |
| Outcome: | The proposed model outperforms existing methods on a benchmark dataset with 39K posts on 3K posts. |
DEER: Descriptive Knowledge Graph for Explaining Entity Relationships (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge graphs lack two desired features for modeling entity relationships: openness and informativeness. |
| Approach: | They propose a self-supervised learning method to extract relation descriptions with the analysis of dependency patterns and generate relation descriptions using a transformer-based relation description synthesizing model. |
| Outcome: | The proposed system extracts and generates high-quality relation descriptions without human labeling. |
The MARCELL Legislative Corpus (2020.lrec-1)
Copied to clipboard
Tamás Váradi, Svetla Koeva, Martin Yamalov, Marko Tadić, Bálint Sass, Bartłomiej Nitoń, Maciej Ogrodniczuk, Piotr Pęzik, Verginica Barbu Mititelu, Radu Ion, Elena Irimia, Maria Mitrofan, Vasile Păiș, Dan Tufiș, Radovan Garabík, Simon Krek, Andraz Repar, Matjaž Rihtar, Janez Brank
| Challenge: | MARCELL corpus provides a rich and valuable source for further studies and developments in machine learning, cross-lingual terminological data extraction and classification. |
| Approach: | They present the results of the project MARCELL CEF Telecom . they aim to collect and deeply annotate a large comparable corpus of legal documents . |
| Outcome: | The MARCELL corpus includes 7 monolingual sub-corpora containing the body of respective national legislative documents. |
Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing algorithms for learning unimodal vision-only or language-only tasks are limited by the size and computational load of fine-tuning large-scale pre-trained neural networks. |
| Approach: | They propose a transformer-based CL architecture for learning bimodal vision-and-language tasks by increasing the number of the learnable parameters dynamically and using knowledge distillation. |
| Outcome: | The proposed model reaches state-of-the-art on vision-and-language tasks. |
DORE: A Dataset for Portuguese Definition Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Definition modelling (DM) is the task of automatically generating a dictionary definition of a specific word. |
| Approach: | They propose to create a dataset for definition modelling for Portuguese with 100,000 definitions and evaluate several deep learning based DM models on the dataset. |
| Outcome: | The proposed dataset will facilitate research and study of Portuguese in wider contexts. |
RoBoCoP: A Comprehensive ROmance BOrrowing COgnate Package and Benchmark for Multilingual Cognate Identification (2023.emnlp-main)
Copied to clipboard
Liviu Dinu, Ana Uban, Alina Cristea, Anca Dinu, Ioan-Bogdan Iordache, Simona Georgescu, Laurentiu Zoicas
| Challenge: | Existing databases for romance cognates are scattered, incomplete, noisy, or have uncertain availability. |
| Approach: | They propose to use etymological information to identify Romance cognates and borrowings from dictionaries to identify their ethymology. |
| Outcome: | The proposed method achieves 94% accuracy on two pairs of Romance languages. |
FigMemes: A Dataset for Figurative Language Identification in Politically-Opinionated Memes (2022.emnlp-main)
Copied to clipboard
| Challenge: | FigMemes is a dataset for figurative language classification in politically-opinionated memes. |
| Approach: | They propose to use figurative language classification to identify politically-opinionated memes by analyzing their datasets and comparing them to other machine learning models. |
| Outcome: | The proposed dataset includes annotations of six commonly used types of figurative language in politically-opinionated memes and a wide range of topics and visual styles. |
MEANT: Multimodal Encoder for Antecedent Information (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal data is an ideal candidate for multimodal evaluation, but information can exist across time. |
| Approach: | They propose a multimodal encoder for anticedent information and a dataset that consists of price, Tweets, and graphical data. |
| Outcome: | The MEANT model improves performance on baselines by 15% and the textual information affects performance far more than visual information on time-dependent tasks. |
Interpretable Question Answering on Knowledge Bases and Text (P19-1)
Copied to clipboard
| Challenge: | Existing evaluation paradigms for ML based question answering models are lacking . a lack of explanation methods has been proposed for QA models . |
| Approach: | They propose an automatic evaluation paradigm for explanation methods in ML based question answering models . they adapt post hoc explanation methods such as LIME and input perturbation to the model . |
| Outcome: | The proposed evaluation paradigm compares explanation methods with human annotations. |
DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks (2020.emnlp-main)
Copied to clipboard
Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kruengkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si, Chunyan Miao
| Challenge: | Data augmentation techniques are widely used to improve machine learning performance . however, due to the complexity of language, it is difficult to generalize such rules for languages. |
| Approach: | They propose a method to generate high quality synthetic data for low-resource tagging tasks . they use unlabeled data only and unlabelled data plus a knowledge base . |
| Outcome: | The proposed method outperforms baselines on NER, part of speech and target based sentiment analysis tasks. |
A Re-evaluation of Knowledge Graph Completion Methods (2020.acl-main)
Copied to clipboard
| Challenge: | Knowledge Graph Completion (KGC) aims at automatically predicting missing links for large-scale knowledge graphs. |
| Approach: | They propose a protocol to evaluate KGC methods that is robust to handle bias in the model, which can substantially affect the final results. |
| Outcome: | The proposed evaluation protocol is robust to handle bias in the model, which can substantially affect the final results. |
Learning to Correct for QA Reasoning with Black-box LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve reasoning capability of large language models rely on accessibility or require significantly increased train- and inference-time costs. |
| Approach: | They propose a method to improve QA reasoning of large language models in a black-box setting by using a trained adaptation model to perform a seq2seq mapping from the often-imperfect reasonings of the original LLM to the correct or improved reasonings. |
| Outcome: | The proposed approach significantly improves reasoning accuracy across various QA benchmarks compared to the best-performing adaptation baselines. |
Effects of Human Adversarial and Affable Samples on BERT Generalization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Limiting quantities of training data is considered a key impediment to achieving generalizability in machine learning. |
| Approach: | They examine the impact of training data quality, not quantity, on a model’s generalizability by comparing human-adversarial and human-affable training samples. |
| Outcome: | The proposed model performance improves with 10-30% h-adversarial instances in text classification and relation extraction tasks. |
CEFR-based Lexical Simplification Dataset (L18-1)
Copied to clipboard
| Challenge: | Existing tools for lexical simplification are not tailored to language education with word levels and lists of candidates subjective. |
| Approach: | They construct a language dataset for lexical simplification based on CEFR levels . target and candidate words are assigned CEFR-J wordlists and English Vocabulary Profile . |
| Outcome: | The proposed method is based on the common European Framework of References for Languages (CEFR) levels and candidates are selected using an online thesaurus. |
Offensive Video Detection: Dataset and Baseline Results (2020.lrec-1)
Copied to clipboard
| Challenge: | a large number of social media platforms discourage users from publishing offensive content . however, there is no method to detect offensive content on these platforms due to the high volume of publications. |
| Approach: | They propose to use text-based machine learning to detect offensive content on different platforms . they use word embedding with Deep Learning classifiers to perform best results . |
| Outcome: | The proposed methods outperform Classic and Deep Learning classifiers in Portuguese and CNN architectures in other features. |
All That Glitters is Not Gold: A Gold Standard of Adjective-Noun Collocations for German (2020.lrec-1)
Copied to clipboard
| Challenge: | Using the GerCo dataset, we identify adjective-noun collocations in German and compare them with statistical associations measures. |
| Approach: | They present a GerCo dataset of adjective-noun collocations for German, such as alter Freund ‘old friend’ and tiefe Liebe ‘deep love’. |
| Outcome: | The GerCo dataset contains 4,732 positive and negative instances of collocations and covers all 16 semantic classes of adjectives defined in the German wordnet GermaNet. |
Continual Learning in Multilingual Sign Language Translation (2025.naacl-long)
Copied to clipboard
| Challenge: | Despite the low translation quality of sign language, many machine learning approaches are still in its infancy. |
| Approach: | They propose to use continual learning for mul- tilingual SLT to improve translation quality. |
| Outcome: | The proposed methods outperform baseline and fine-tuning approaches in sign language translation. |
“What is on your mind?” Automated Scoring of Mindreading in Childhood and Early Adolescence (2020.coling-main)
Copied to clipboard
Venelin Kovatchev, Phillip Smith, Mark Lee, Imogen Grumley Traynor, Irene Luque Aguilera, Rory Devine
| Challenge: | Existing studies show that children who excel at mindreading are more likely to be identified as popular by classmates and have reciprocated friendships. |
| Approach: | They propose to automate the scoring of mindreading ability in middle childhood and early adolescence using a new corpus of 11,311 question-answer pairs in English from 1,066 children aged from 7 to 14 . |
| Outcome: | The proposed scoring system is based on 11,311 question-answer pairs in English from 1,066 children aged from 7 to 14 . the results demonstrate the applicability of state-of-the-art NLP solutions to a new domain and task. |
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (D18-1)
Copied to clipboard
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica Gašić
| Challenge: | a dataset of 10k human-human written conversations is one order of magnitude larger than previous annotated task-oriented corpora. |
| Approach: | They propose to collect 10k human-human written conversations from a crowd-sourced dataset using crowd-sourcing. |
| Outcome: | The proposed dataset is one order of magnitude larger than previous annotated task-oriented corpora and shows the usability of the data and sets a baseline for future studies. |
Intertextual Correspondence for Integrating Corpora (L18-1)
Copied to clipboard
| Challenge: | Using intertextual correspondence, we can combine annotated text corpora to create new annotation connections. |
| Approach: | They propose to use intertextual correspondence as an integrative technique for combining annotated text corpora. |
| Outcome: | The proposed technique can be used to build argumentative arguments in two annotated text corpora. |
HateGAN: Adversarial Generative-Based Data Augmentation for Hate Speech Detection (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to detect online hate speech depend heavily on labeled datasets for training, which results in poor detection performance of the hate speech class. |
| Approach: | They propose a deep generative reinforcement learning model which augments two commonly-used hate speech detection datasets with the HateGAN generated tweets. |
| Outcome: | The proposed model improves the detection performance of hate speech class regardless of the classifiers and datasets used in the detection task. |
Creating a Dataset for Named Entity Recognition in the Archaeology Domain (2020.lrec-1)
Copied to clipboard
| Challenge: | Currently, there is no way to find 'by-catch', single finds of a different type, in the metadata of excavation reports. |
| Approach: | They propose to train NER classifiers on Dutch excavation reports to help archaeologists find structured information in archaic documents. |
| Outcome: | The proposed dataset contains 31k annotations between six entity types (artefact, time period, place, context, species & material). |
Characterizing Similarities and Divergences in Conversational Tones in Humans and LLMs by Sampling with People (2024.acl-long)
Copied to clipboard
| Challenge: | Existing taxonomies or text corpora suffer from experimenter bias and are not representative of real-world distributions. |
| Approach: | They propose an iterative method for simultaneously eliciting conversational tones and sentences . they run 50 iterations with human participants and GPT-4 and obtain a dataset of sentences and frequent conversational tone. |
| Outcome: | The proposed method can be used to characterize the differences between humans and LLMs. |
Back-Training excels Self-Training at Unsupervised Domain Adaptation of Question Generation and Passage Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Using self-training to train unsupervised domains can be expensive, resulting in poor generalization due to distributional shift. |
| Approach: | They propose to use unaligned data to train unsupervised domain adaptation models using cheap synthetically generated labeled data. |
| Outcome: | The proposed method significantly outperforms self-training on question generation and passage retrieval domains and on MLQuestions and PubMedQA. |
DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions (2023.acl-long)
Copied to clipboard
| Challenge: | Modern machine learning relies on datasets to develop and validate research ideas. |
| Approach: | They propose a dataset recommendation system that uses a training set and an evaluation set to help people find relevant datasets. |
| Outcome: | The proposed model finds more relevant search results than existing third-party search engines. |
Exploring BERT-Based Classification Models for Detecting Phobia Subtypes: A Novel Tweet Dataset and Comparative Analysis (2024.lrec-main)
Copied to clipboard
| Challenge: | Phobias are characterized by an intense and irrational fear of specific objects, situations, or activities despite there being no real risk or only a minor threat involved. |
| Approach: | They propose to use a dataset of 811,569 English tweets from user timelines spanning 102 phobia subtypes over six months to classify users into 65 specific phobias. |
| Outcome: | The proposed dataset includes 47,614 self-diagnosed phobia users and a high f1 score for binary classification and multi-class classification. |
An Empirical Examination of Online Restaurant Reviews (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for opinion mining and sentiment analysis focus on extracting either positive or negative opinions from texts and determining the targets of these opinions. |
| Approach: | They propose a corpus-based scheme that detects evaluative language at a finer-grained level. |
| Outcome: | The proposed scheme classifies each sentence into one of four evaluation types based on the proposed scheme. |
Chunk Different Kind of Spoken Discourse: Challenges for Machine Learning (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing chunkers for spoken data are based on a corpus composed of monologues and spontaneous talk in interaction. |
| Approach: | They propose to use CRFs to develop a chunker for spoken data . the chunker is based on a small corpus composed of two kinds of discourse . |
| Outcome: | The proposed chunker is based on a spoken corpus composed of monologue and spontaneous talk in interaction. |
iSign: A Benchmark for Indian Sign Language Processing (2024.findings-acl)
Copied to clipboard
Abhinav Joshi, Romit Mohanty, Mounika Kanakanti, Andesha Mangla, Sudeep Choudhary, Monali Barbate, Ashutosh Modi
| Challenge: | Indian Sign Language has limited resources for developing machine learning and data-driven approaches for automated language processing. |
| Approach: | They propose to use a sign language dataset to provide a benchmark for Indian Sign Language processing. |
| Outcome: | The proposed benchmarks will help improve sign language translation models and open up various ways for advancing natural language processing. |
The Use of Text Alignment in Semi-Automatic Error Analysis: Use Case in the Development of the Corpus of the Latvian Language Learners (L18-1)
Copied to clipboard
| Challenge: | Using error annotation methods, the corpus of the Latvian language learners can be adapted for other languages with relatively free word order. |
| Approach: | They propose a method for creating error annotated corpora using text correction, automated morphological analysis, automated text alignment and error annotation. |
| Outcome: | The proposed method has been approbated in the development of the corpus of the Latvian language learners. |
PerCQA: Persian Community Question Answering Dataset (2022.lrec-1)
Copied to clipboard
| Challenge: | Community Question Answering (CQA) forums provide answers to many real-life questions. |
| Approach: | They propose to make Persian dataset PerCQA public to encourage more research in Persian CQA. |
| Outcome: | The proposed dataset contains 989 questions and 21,915 annotated answers from the most well-known Persian forum. |
An Efficient Retrieval-Based Method for Tabular Prediction with LLM (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for tabular prediction rely on extensive pre-training or fine-tuning of LLMs . a retrieval-based approach eliminates the need for training any modules or performing data augmentation . |
| Approach: | They propose a retrieval-based approach that utilizes the powerful capabilities of large language models in representation, comprehension, and inference. |
| Outcome: | The proposed method exhibits strong predictive performance on tabular prediction task, affirming its practicality and effectiveness. |
REPROLANG 2020: Automatic Proficiency Scoring of Czech, English, German, Italian, and Spanish Learner Essays (2020.lrec-1)
Copied to clipboard
| Challenge: | a new paper aims to reproduce the work described in Vajjala & Rama (2018) . the paper focuses on features-based and neural approaches to essay scoring in Czech, German and Italian . |
| Approach: | They propose to replicate the work described in Vajjala & Rama 2018, ‘Experiments with universal CEFR classification’, as part of REPROLANG 2020. |
| Outcome: | The proposed methods perform better than feature-based models for large text datasets, though neural network modifications do bring performance closer to the best feature-driven models. |
AxCell: Automatic Extraction of Results from Machine Learning Papers (2020.emnlp-main)
Copied to clipboard
Marcin Kardas, Piotr Czapla, Pontus Stenetorp, Sebastian Ruder, Sebastian Riedel, Ross Taylor, Robert Stojnic
| Challenge: | AxCell is an automatic machine learning pipeline for extracting results from papers . it uses a table segmentation subtask to learn relevant structural knowledge that aids extraction. |
| Approach: | They propose to use a table segmentation subtask to learn relevant structural knowledge that aids extraction. |
| Outcome: | The proposed approach improves state of the art for results extraction and can be used for semi-automated results extraction in production. |
The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | a paper argues that human label variation impacts all stages of the ML pipeline . human label variations are often considered noise due to disagreement, subjectivity in annotation or multiple plausible answers. |
| Approach: | They propose to reconcile different notions of human label variation and propose a repository of publicly-available datasets with un-aggregated labels. |
| Outcome: | The proposed approaches are compared with publicly available datasets with un-aggregated labels and identify gaps. |
RExBench: Can coding agents autonomously implement AI research extensions? (2026.acl-long)
Copied to clipboard
| Challenge: | Existing large language model (LLM) agents are not capable of performing research extension tasks autonomously. |
| Approach: | They propose a benchmark to evaluate LLM agents' ability to extend existing AI research . they use extensions of 12 recently published research papers accompanied by domain expert-written instructions . |
| Outcome: | The proposed benchmark evaluates 12 LLM agents implemented using aider and OpenHands. |
HelloFresh: LLM Evalutions on Streams of Real-World Human Editorial Actions across X Community Notes and Wikipedia edits (2024.findings-acl)
Copied to clipboard
Tim Franzmeyer, Aleksandar Shtedritski, Samuel Albanie, Philip Torr, Joao F. Henriques, Jakob Foerster
| Challenge: | a better understanding of LLM capabilities on real world tasks is vital for safe development and deployment. |
| Approach: | They propose a new LLM called HelloFresh that uses real-world data to measure performance . they backtest the model and find it yields a temporally consistent ranking . |
| Outcome: | The proposed benchmarks outperform static evaluation data and test data on Wikipedia pages. |
MLAlgo-Bench: Can Machines Implement Machine Learning Algorithms? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, the top-performing models achieve a 48.8% task completion rate on realizing machine learning algorithms . |
| Approach: | They propose a benchmark to test machine learning's ability to generate ML code for humans . they propose an automatic evaluation framework with metrics such as task pass rate and time overhead . |
| Outcome: | The proposed benchmark is unique in its focus on interpreting complex human instructions and producing multi-step, high-complexity code. |
MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing pretrained language models for mental health detection are inadequate . one in four people worldwide suffers from mental disorders . |
| Approach: | They train and release two pretrained masked language models to benefit machine learning for mental healthcare research . they demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks. |
| Outcome: | The proposed models improve mental health detection tasks on several benchmarks and are available for free. |
Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Mainstream of automatic speech recognition (ASR) has shifted from pipeline methods to end-to-end (E2E) methods. |
| Approach: | They propose to integrate a pre-trained speech representation model and a large language model (LLM) for automatic speech recognition in an end-to-end manner. |
| Outcome: | The proposed model achieves comparable performance to modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach. |
A new European Portuguese corpus for the study of Psychosis through speech analysis (2022.lrec-1)
Copied to clipboard
| Challenge: | Psychosis is a clinical syndrome characterized by symptoms such as hallucinations, delusions, thought disorders and disorganized speech. |
| Approach: | They describe the creation of the first European Portuguese corpus for the identification of the presence of speech characteristics of psychosis. |
| Outcome: | The results show that spontaneous speech presents more identifiable characteristics than read speech to differentiate healthy and patients diagnosed with psychosis. |
A Survey of Deep Learning for Mathematical Reasoning (2023.acl-long)
Copied to clipboard
| Challenge: | a survey of deep learning for mathematical reasoning examines the field . a comprehensive reading list is provided to assist readers interested in the field. |
| Approach: | They present a survey of deep learning for mathematical reasoning over the past decade . they outline directions for future research and highlight potential for further exploration . |
| Outcome: | The proposed framework is based on the results of a decade-long survey of deep learning for mathematical reasoning. |
A Federated Approach to Predicting Emojis in Hindi Tweets (2022.emnlp-main)
Copied to clipboard
| Challenge: | emojis are a visual modality to, often private, textual communication, but their use tends to cluster into the frequently used and the rarely used eojis. |
| Approach: | They propose to use 118k tweets to predict emojis in Hindi and a federated learning algorithm to achieve a balance between model performance and user privacy. |
| Outcome: | The proposed approach achieves comparative scores with more complex centralised models while minimising risks to user privacy. |
LEGOBench: Scientific Leaderboard Generation Benchmark (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a growing number of papers make it difficult to stay informed about the latest state-of-the-art research. |
| Approach: | They propose a benchmark to evaluate systems that generate scientific leaderboards . they use 22 years of submission data on arXiv and 11k machine learning leaderboard data on paperswithcode . |
| Outcome: | The proposed model shows significant performance gaps in the LEGOBench model . the model is based on a language model and four graph-based leaderboard generation task configuration . |
FSTs vs ICL: Generalisation in LLMs for an under-resourced language (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLMs have been widely adopted to tackle many traditional NLP tasks, but their effectiveness remains uncertain in scenarios where pre-trained models have limited prior knowledge of a language. |
| Approach: | They propose a rule-based method using a finite-state transducer and an in-context learning method that provides the model with string transduction examples. |
| Outcome: | The proposed method outperforms FSTs in zero-shot settings while ICL surpasses FLMs. |
Analyzing Transformers in Embedding Space (2023.acl-long)
Copied to clipboard
| Challenge: | Understanding Transformer-based models has attracted significant attention . a zero-pass approach is feasible for some parameters, and for two-layer attention networks . |
| Approach: | They propose a theoretical framework where parameters of a trained Transformer are interpreted by projecting them into the embedding space. |
| Outcome: | The proposed framework shows that pre-trained and fine-tuned models can be interpreted in embedding space. |
MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classification (2024.emnlp-main)
Copied to clipboard
| Challenge: | a novel dataset of text-embedded images associated with the LGBTQ+ Pride movement is presented in this paper . a new framework for analyzing text-based images is proposed to address this challenge . |
| Approach: | They propose a new dataset for machine learning that includes hate, targets of hate, stance, humor and a framework for efficient downstream learning while preserving the knowledge of the pre-trained CLIP model. |
| Outcome: | The proposed framework achieves superior performance on two real-world datasets. |
Data Interpreter: An LLM Agent for Data Science (2025.findings-acl)
Copied to clipboard
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Robert Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Yongxin Ni, Zhibin Gou, Zongze Xu, Yuyu Luo, Chenglin Wu
| Challenge: | Large Language Models (LLMs) excel in various domains but face challenges when applied to data science workflows due to their complex, multi-stage nature. |
| Approach: | They propose a hierarchical graph-based agent that represents complexity and a progressive strategy for step-by-step verification, refinement, and consistent context management. |
| Outcome: | The proposed agent surpasses state-of-the-art baselines on the MATH dataset and performs better on InfiAgent-DABench. |
Multi-Channel Spatio-Temporal Transformer for Sign Language Production (2024.lrec-main)
Copied to clipboard
| Challenge: | Sign language production models ignore structural correlations between channels and use multi-channel spatial attention to capture correlations across channels. |
| Approach: | They propose a novel approach to transform sign language into a unified feature representation using multi-channel spatial attention and temporal attention to learn sequential dependencies for each channel over time. |
| Outcome: | The proposed model outperforms state-of-the-art models on two sign language datasets from diverse cultures. |
Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition (2024.lrec-main)
Copied to clipboard
Yash Jain, David M. Chan, Pranav Dheram, Aparna Khare, Olabanji Shonibare, Venkatesh Ravichandran, Shalini Ghosh
| Challenge: | Existing methods for pre-training for automatic speech recognition (ASR) focus on single-stage pre-train followed by fine-tuning on downstream task. |
| Approach: | They propose a multi-modal pre-training method that combines unsupervised pre-training with translation-based supervised mid-training. |
| Outcome: | The proposed method improves WERs by 38.45% over baselines on both Librispeech and SUPERB. |
VER: Unifying Verbalizing Entities and Relations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new model for verbalizing entities and relations is proposed to help understand entities and relationships . a unified model for Verbalizing Entities and Relations is proposed . |
| Approach: | They propose a model that takes any entity or entity set as input and generates a sentence to represent entities and relations. |
| Outcome: | The proposed model can generate sentences describing entities and relations . it can be used to explain entities and relationships, and to perform commonsense reasoning tasks . |
Uniform Information Density and Syntactic Reduction: Revisiting *that*-Mentioning in English Complement Clauses (2025.emnlp-main)
Copied to clipboard
| Challenge: | Uniform Information Density (UID) hypothesis suggests that speakers exploit this variability to maintain a consistent rate of information transmission during language production. |
| Approach: | They propose that speakers exploit this variability to maintain a consistent rate of information transmission during language production. |
| Outcome: | The proposed hypothesis replicates the established relationship between information density and *that*-mentioning . |
PolyNERE: A Novel Ontology and Corpus for Named Entity Recognition and Relation Extraction in Polymer Science Domain (2024.lrec-main)
Copied to clipboard
| Challenge: | a new ontology for polymer-relevant entities and relations is available for training data . the ontologies are customizable to adapt to specific research needs. |
| Approach: | They propose a polymer-relevant ontology featuring crucial entities and relations . the ontologies are customizable to adapt to specific research needs . |
| Outcome: | The proposed ontology can extract polymer-relevant information from scientific papers . it can be customized to adapt to specific research needs . |
Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Building socially-intelligent AI agents involves creating agents that can sense, perceive, reason about, learn from, and respond to affect, behavior, and cognition of other agents. |
| Approach: | They propose a set of technical challenges and open questions for researchers to advance Social-AI. |
| Outcome: | The proposed frameworks are based on the social intelligence competencies that evolved over thousands of years in Homo sapiens and are expected to be the foundations for the development of social-intelligent AI agents. |
Rebalancing Label Distribution While Eliminating Inherent Waiting Time in Multi Label Active Learning Applied to Transformers (2024.lrec-main)
Copied to clipboard
| Challenge: | Data annotation is a resourceintensive endeavor, necessitating human involvement and expertise. |
| Approach: | They propose to annotate instances to rebalance label distribution by judiciously selecting and limiting the data to be annotated. |
| Outcome: | The proposed method mitigates biases, improves model performance and reduces strategy-dependent disparities. |
Embedding-Converter: A Unified Framework for Cross-Model Embedding Transformation (2025.acl-long)
Copied to clipboard
| Challenge: | Embedding models are fundamental to modern machine learning, but the continuous development of new models presents a major challenge. |
| Approach: | They propose a framework for efficiently transforming embeddings between different models, thus avoiding costly ‘re-embedding’. |
| Outcome: | The proposed framework achieves 100 times faster and cheaper computations in real-world applications. |
Analyze Like a Venture Capitalist: Information-Gain and Knowledge Enhanced Graph Reasoning for Startup Success Prediction (2026.findings-acl)
Copied to clipboard
| Challenge: | Most venture capital investments fail, while a few deliver outsized returns. |
| Approach: | They propose a framework that synthesizes relational evidence across sources . they propose combining information-gain-driven retriever and knowledge base to ground reasoning . |
| Outcome: | The proposed framework achieves +5.9% F1 and +22.1% Precision@5 over state-of-the-art baselines. |
A Position Paper on the Automatic Generation of Machine Learning Leaderboards (2025.emnlp-main)
Copied to clipboard
| Challenge: | Automated leaderboard generation is a tool for comparing prior work with a tabular overview of experimental results. |
| Approach: | They propose an automatic leaderboard generation framework to standardise how the task is defined. |
| Outcome: | The proposed framework standardises how the ALG task is defined and proposes new directions . the proposed framework includes recommendations for datasets and metrics that promote fair evaluation . |
AutoCT: Automating Interpretable Clinical Trial Prediction with LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Clinical trials are expensive and time-consuming, and accurate trial prediction is key to advancing medical treatments. |
| Approach: | They propose a framework that combines reasoning capabilities of large language models with the explainability of classical machine learning to generate, evaluate, and refine tabular features without human input. |
| Outcome: | The proposed framework performs better than SOTA methods on clinical trial prediction tasks within a limited number of iterations. |
Introducing Verification Task of Set Consistency with Set-Consistency Energy Networks (2025.acl-long)
Copied to clipboard
| Challenge: | Traditional methods that rely on 1:1 pairwise comparisons fail to capture inconsistencies . few studies have addressed related problems in the domain of factual inconsistenency detection . |
| Approach: | They propose a set-consistency verification task that assesses logical coherence of entire sets . they propose 'set-consistent energy network' that employs a margin-based loss to learn the compatibility among a collection of statements . |
| Outcome: | The proposed model outperforms existing methods and significantly outperformed existing models. |
ReDepress: A Cognitive Framework for Detecting Depression Relapse from Social Media (2025.emnlp-main)
Copied to clipboard
Aakash Kumar Agarwal, Saprativa Bhattacharjee, Mauli Rastogi, Jemima S. Jacob, Biplab Banerjee, Rashmi Gupta, Pushpak Bhattacharyya
| Challenge: | Almost 50% of depression patients face the risk of going into relapse. |
| Approach: | They propose to validate a social media dataset on depression relapse using cognitive theories of depression. |
| Outcome: | The first clinically validated social media dataset focused on depression relapse comprises 204 Reddit users annotated by mental health professionals. |
CircuitSynth: Reliable Synthetic Data Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches lack mechanisms to balance linguistic expressivity with formal guarantees regarding validity and coverage. |
| Approach: | They propose a neuro-symbolic framework that decouples semantic reasoning from surface realization. |
| Outcome: | The proposed framework achieves 100% Schema Validity even in complex logic puzzles where unconstrained baselines fail (12.4%) while outperforming state-of-the-art methods in rare-combination coverage. |
A Counterfactual Explanation Framework for Retrieval Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing literature on explainability of information retrieval has focused on illustrating the concept of relevance concerning a retrieval model. |
| Approach: | They propose to add terms to a document to improve its ranking to answer the question of which words played a role in not being favored by a retrieval model. |
| Outcome: | The proposed framework predicts counterfactuals for statistical and deep-learning models. |
From Words to Pixels: A Comprehensive Survey on Large Language Models in Visual Segmentation (2026.acl-long)
Copied to clipboard
| Challenge: | Visual segmentation with instruction has been a challenging task for many years . large language models and large multimodal models have spurred a new wave of research . |
| Approach: | They review recent works in LLM-based visual segmentation and analyze their architectural innovations, training strategies, and benchmark performance. |
| Outcome: | The present study reviews the most recent works in LLM-driven visual segmentation . it identifies key challenges and promising future directions . |